Big Data, Hadoop
and MapReduce

Scale, storage and distributed processing

Liu Junhua, PhD
j@forth.ai

Topic 1: what you should be able to do

Use these outcomes to check your understanding of Topic 1.

SkillExplain or demonstrate
DiagnoseUse the five Vs; distinguish data quality from capacity.
CompareScale up vs scale out; explain a benefit and a cost.
ExplainHDFS storage, YARN resources and MapReduce processing.
TraceRead/write paths, task allocation and keyed aggregation.
JustifyStorage choice, combiner correctness and EMR data survival.
02 / 80

Topic 1 study map

Study a section, answer without looking, then check the worked explanation.

TopicRevision taskSlides
1 · Big Data + scalingDiagnose pressures; compare capacity choices.8–26
2 · Hadoop + HDFSSeparate roles; trace storage and recovery.27–41
3 · YARNMatch each actor to its scope and actions.42–48
4 · MapReduceTrace pairs, groups, tasks and output files.49–65
5 · EMR + conclusionExplain node roles; practise and check answers.66–80
03 / 80

Five lectures: from distributed data to useful results

01HadoopHDFS, YARN andMapReduceDistribute storageand work.02SparkArchitecture andDataFramesBuild a datapipeline.03PerformancePlans, shuffle andstorageExplain whereresources go.04Spark MLFeatures, modelsand evaluationEvaluatepredictionscorrectly.05HiveCatalog, tablesand SQLQuery shared data.
04 / 80

From sales data to a demand forecast

105HadoopStore15SparkPrepare18Spark MLPredict18HiveQueryPerformance · tune the whole pipeline
05 / 80

Practise the concepts in five labs

06 / 80

Real-world example: turning web logs into user analytics

Store the records, share the work, and check the answer against the business question.

Analytics goalHow many requestsused each page?retainStorageKeep the growing logfilesprocessProcessingCount requestsacross machinesverifyReportCheck counts andsave results
07 / 80

Big Data

What makes data useful, and when does its size become a problem?

08 / 80

Big Data: size matters relative to the task

Describe the data and the task before choosing tools.

QuestionMeaningExample
What makes it big?Data strains the available storage or processing.No universal file-size threshold.
What makes it useful?It can answer a defined question.Orders measure sales; clicks measure interest.
What makes it trustworthy?Identity, missing values and timing are checked.A Tuesday release cannot inform Monday’s prediction.
09 / 80

The five Vs describe different pressures

VolumeHow muchmust fit?VelocityHow quickly mustit arrive?VarietyWhich formatsand sources?VeracityHow reliableis it?ValueWhich decisionimproves?
10 / 80

Volume: the amount of data

Volume is the quantity of data to store and process. It becomes a challenge when the dataset exceeds the capacity of the available system.

Partitioning divides the work; it does not reduce the total data. Here, all six records remain across three partitions.

Before · one datasetPartition by record rangeAfter · three partitionsTask A · 2 recordsR1R2Task B · 2 recordsR3R4Task C · 2 recordsR5R66 input records → 3 partitions × 2 records. No records are lost.R1R2R3R4R5R6
Illustrative data transformation11 / 80

Velocity: the rate of incoming data

Velocity describes how quickly data arrives and must be processed. A system must keep up with arrivals and meet the required response time.

Arrivals exceed processing by one event per interval, so the backlog grows. Buffering delays the problem; sustained throughput must catch up.

Arrivals · 4 per intervalQueueCompleted · 3 per intervalInterval 1Interval 2First in → first outE1E1E2E2E3E3E4E4E5E5E6E6E7E7E8E8
Illustrative data transformation12 / 80

Variety: different forms of data

Variety means differences in data formats, structures and meaning. Tables, nested records and images require different ways to read and interpret them.

Parsing makes fields usable, but does not make their meanings equivalent. Image dimensions describe its size, not what the image shows.

Source dataUsable fieldsCSVid,amount7,12Read CSVNumeric fieldsid: 7 · amount: 12JSON{"id":7,"event":{"type":"view"}}Parse JSONEvent fieldsid: 7 · type: viewIMAGEI7Measure width and heightImage metadataI7 · width: 2 · height: 2The image produces dimensions, not a semantic label.
Illustrative data transformation13 / 80

Veracity: the trustworthiness of data

Veracity concerns data accuracy, completeness and reliability. Duplicates, missing values and incorrect records can make a technically correct calculation misleading.

Exclude the confirmed replay and keep the missing amount unknown. The accepted amount is 10; it is not a complete total for all orders.

Incoming recordsRecords with statusOrder 8 · user u1Amount: 10Check order IDACCEPT · order 8Amount: 10 · count onceOrder 8 · user u1Amount: 10 · replayConfirm the replayEXCLUDE · order 8Confirmed replayOrder 9 · user u2Amount: ?Check amountREVIEW · order 9Amount remains unknownKnown accepted amount: 10. Order 9 still needs review.
Illustrative data transformation14 / 80

Value: the usefulness of data

Value is the benefit gained by using data to answer a meaningful question or improve a decision. More data or faster processing does not automatically create value.

Demand of 7 minus stock of 4 suggests a gap of 3. The uncertainty range makes this evidence for a replenishment decision, not a guaranteed order quantity.

1Read the data2Find the gap3Review replenishmentExpected demand · 7 unitsDemand range: 5–9 unitsAvailable stock · 4 unitsRequired arrival: Friday7 − 4 = 3 units4 covered3 shortGap range: 1–5 unitsDemand range minus stockReview: 3 unitsRange: 1–5 unitsArrival: FridayReview before placing an order.Store N · item A | Each box represents one unit.
Illustrative data transformation15 / 80

Scaling

How can we add capacity when one machine is no longer enough?

16 / 80

Compare scaling up with scaling out

More machines introduce coordination, networks and partial failures.

Scale upScale outBenefit and costAdd resources to one host.CPU, memory and storageAdd hosts and divide the work.Tasks run on separate machines.Parallel workCoordinationMore capacity, with time spentmoving and combining results.
17 / 80

Scale up: add resources to one machine

A larger server helps only when the added resource is the bottleneck.

Host A · beforeAdd resourcesHost A · upgradedCPUMemoryStorageOne machine before and after. One failure domain.
Illustrative resource upgrade18 / 80

Scale out: divide work across machines

The benefit of parallel work must exceed scheduling and communication costs.

Partition the valuesProcess in parallelCombine partial sumsP11 + 2 + 3Worker 1Sum: 6P24 + 5 + 6Worker 2Sum: 15P37 + 8 + 9Worker 3Sum: 24Total6 + 15 + 24= 45
19 / 80

Trade-off: a shared result needs coordination

Workers process data in parallel, but they must exchange partial results to produce one correct answer.

Count locallyExchange by keyOne combined answerWorker Ared: 2 · blue: 1Worker Bred: 1 · green: 1red2 + 1blue1green1red: 3blue: 1green: 1
20 / 80

Revision: identify the problem before the remedy

For a scenario answer: identify the pressure → explain the cause → justify the remedy.

DistinctionRuleReason
Volume / velocityAmount of data / rate of arrival.A growing queue signals insufficient throughput.
Variety / veracity / valueFormats / trust / useful outcome.Different pressures need different remedies.
Scale upAdd CPU, memory or storage to one host.Helps only if that resource limits the work.
Scale outAdd hosts and divide work.Communication and coordination reduce the gain.
Capacity / correctnessMeasure speed; validate meaning separately.More workers can still produce a wrong answer.
21 / 80

Case study

How can we produce a correct hourly report before the next batch arrives?

22 / 80

Case study: an hourly report is late and wrong

Count distinct posts per account within 60 minutes. Diagnose the incorrect count and the missed deadline.

Hourly inputOne batch jobA report with two problemsPost eventsP1, P2, P1 replayAttachmentI1 belongs to P1Account rowsU1, U1 repeatRead, join, countNo duplicate checks90 minutes per batchU1: 6 rowsIncorrect post countLate by 30 minutesDeadline: 60 minutes
23 / 80

Remove confirmed duplicates before the join

Remove the replay of P1 and the repeated account row, then join two posts to one account record.

Before validationRemove confirmed repeatsAfter validationP1, P2, P1 replayKeep one valid recordfor each source key.P1, P2U1, U1 repeatKeep one valid recordfor each source key.U1Naive join: 3 × 2 = 6 rowsValidated join: 2 × 1 = 2 rows
24 / 80

The queue grows when a batch takes too long

Batches arrive every 60 minutes, but each takes 90 minutes, so their waiting times grow from 0 to 30 to 60 minutes.

MinutesWaitingProcessing06090120180270Batch 1Wait: 0 min90 minBatch 2Wait: 30 min90 minBatch 3Wait: 60 min90 minArrival every 60 min < processing time of 90 min
25 / 80

Accept the design only after these checks pass

The report must show two posts for U1, resist replay, and finish within the 60-minute deadline.

Correct countReference: P1 and P2Required result: U1 = 2Replay safetySend P1 again.Required result: still 2DeadlineInclude all processing steps.Finish within 60 minutes.
26 / 80

Hadoop architecture

How does Hadoop divide storage, resource management and processing?

27 / 80

Hadoop makes a cluster work together

Apache Hadoop is open-source software for distributed storage and processing across a cluster of computers.

Data filesHadoop clusterResultOne datasetABCCoordination servicesWorker 1AWorker 2BWorker 3CSaved outputEach worker can store data and run tasks.
28 / 80

Follow one job from stored data to a saved result

HDFS stores the input and output. YARN allocates resources. MapReduce tasks count the records.

01HDFSStore the input02YARNAllocate resources03MapReduceProcess the records04HDFSStore the result4 recordsR1 · R2R3 · R4Two input blocksCPUMemoryResources for taskson worker machinesMap task A2Map task B2Reduce: 2 + 2 = 4Record count4Store input → allocate resources → process records → save output
29 / 80

One job connects storage, resources and processing

YARN allocates resources. MapReduce tasks read data from HDFS, process it, and save results to HDFS.

Hadoop clusterClientYour analysis jobYARNShares CPU + memory across applicationsMapReduce applicationMap and reduce tasks run in containers on worker machines.HDFSInput filesResult filesSubmit jobRequest CPU + memoryAllocate containersRead inputWrite resultsJob / resourcesFile data
30 / 80

Hadoop separates storage, resources and processing

These components work together. Each component solves a different problem.

HDFSDistributed file storageYARNCluster resource allocationMapReduceBatch processing framework
31 / 80

HDFS

Store blocks. Track locations. Recover replicas.

32 / 80

HDFS separates the file directory from its bytes

The NameNode tracks files and block locations; DataNodes store the block replicas.

Hadoop Distributed File System (HDFS)One logical file/logs/day.txtBlock ABlock BNameNode · the directoryFile → blocks → DataNode locationsMetadata describes the file.DataNodes · the storage workersDN1: A, B · DN2: A, BReplicas hold the actual file bytes.
33 / 80

HDFS splits a file into blocks and stores replicas

This example stores two copies of each block on DataNodes, while the NameNode records their locations.

Logical fileDataNodesBlock reportsNameNode catalogBlock dataABDN1DN2DN3AABBDN1: ADN2: A, BDN3: BBlock → locationsA → DN1, DN2B → DN2, DN3ABABADN1: AA, BDN2: A, BBDN3: B
Illustrative data transformation34 / 80

Clients obtain metadata, then read block bytes

Clients get block locations from the NameNode, then read the data directly from DataNodes.

ClientReads block AANameNodeBlock locationsDataNode DN2Stored block bytesA → DN2A1 · Request block locations?A?2 · Return location metadataA → DN23 · Request the blockRead A4 · Return the block bytesBlock A
Illustrative data transformation35 / 80

A write uses a replication pipeline

After the NameNode selects DataNodes, the client streams block data through the pipeline to create three replicas.

ClientFile FBlock dataANameNodeCreates file metadataSelects the pipeline1 · Create file and request a block?Create F2 · Return placement metadataDN1 → DN2 → DN33 · Stream block A through the replication pipelineDN1ADN2ADN3A
Illustrative data transformation36 / 80

A failed replica changes the read path

The client reads a surviving replica while the NameNode arranges a new copy to restore replication.

ClientANameNodeMonitors replica locationsDN1UnavailableDN2Surviving replicaDN3Repair targetAAA1 · Read fails2 · Retry DN23 · Detect lossNo heartbeatSchedule repairCopy A to DN34 · Copy A
Illustrative data transformation37 / 80

HDFS supports large sequential workloads

HDFS suits large sequential reads. Combine small records into larger files to reduce file metadata overhead.

Good fitLarge filesRead many records in sequence.Use for batch analysis.Small-file pressureMetadataMany tiny filesEach file needs metadata.Small reads add overhead.Design responseCombine recordsWrite larger analytics files.Keep records accessible inside.
38 / 80

Case study: choose storage for three workloads

5 min · Your team must store these three workloads. Choose the best HDFS fit, explain the limits of the other two, and propose suitable storage designs.

ADaily log archives20 files · 5 GB eachScan all records each night.Keep closed archives unchanged.BProduct images10 million files · 50 KB eachFetch one image per request.Serve images to an online shop.CLive user profilesSmall records · frequent editsRead and update on each click.Respond quickly to users.
39 / 80

Solution: use HDFS for the log archives

Use HDFS for batch scans, object storage for image retrieval, and a database for frequent profile updates.

ADaily log archivesHDFS is a strong fit.Large sequential batch readsFew files limit metadata overhead.BProduct imagesUse object storage for serving.Tiny files add HDFS metadata.For batch analysis, pack records.CLive user profilesUse a database for updates.HDFS is not a record-update API.Keep analytics exports separate.
40 / 80

HDFS revision: roles, paths and suitability

Explain both the mechanism and its consequence; a component name alone is incomplete.

QuestionAnswerWhy it matters
Who stores what?NameNode: metadata. DataNodes: block bytes.Metadata lookup and data transfer are separate.
How does a read work?Client asks NameNode, then reads a DataNode.File bytes do not pass through the NameNode.
How does a write work?NameNode selects nodes; client streams a pipeline.Replicas store copies of the same block.
What if a replica fails?Read a survivor; restore the missing replica.Replication supports availability after node loss.
Which workload fits?Large sequential scans; fewer larger files.Tiny files add metadata and file-open overhead.
41 / 80

YARN

Allocate resources. Coordinate applications.

42 / 80

YARN shares cluster resources among applications

The ResourceManager grants resources; each ApplicationMaster coordinates its job through worker NodeManagers.

ResourceManagerCluster-wide allocatorAvailable CPU + memory2 · Grant resources1 · Request resourcesWorker 1NodeManagerLaunch + monitor locallyWorker 2NodeManagerLaunch + monitor locallyApplicationMasterIn its own containerTask containerTask containerCPU + memory3 · Ask NodeManagers to launch tasks in granted containersYet Another Resource Negotiator
43 / 80

YARN revision: actor, scope and responsibility

YARN allocates resources; the application’s tasks perform the calculation.

ActorScopeResponsibility
ResourceManagerWhole clusterGrants resources according to capacity and policy.
ApplicationMasterOne applicationRequests resources; coordinates and tracks work.
NodeManagerOne worker machineLaunches and monitors local containers.
ContainerOne resource allocationCPU and memory for a process; not necessarily Docker.
44 / 80

First, YARN starts the application coordinator

The ApplicationMaster needs its own container before requesting task resources.

ClientSubmit applicationResourceManagerAccept the request.Allocate an AMcontainer.NodeManagerLaunchApplicationMasterApplicationMasterRegister with RMsubmitstart AMlaunch
45 / 80

Then, the coordinator launches and tracks work

Tasks finish. YARN releases their resources. The application unregisters.

ApplicationMasterRequest taskcontainersResourceManagerGrant availableallocationsNodeManagersLaunch work incontainersApplicationMasterTrack completion.Unregister.requestallocatelaunch requeststaskprogress
46 / 80

Self-check: who allocates, launches and calculates?

Answer before moving to the worked explanation.

SituationYour task
A word-count application needs two task containers.Name the actor that requests them and the actor that grants them.
The granted containers are on two worker machines.Who launches the work? Who tracks application progress?
A reducer produces the final word counts.Does the ResourceManager calculate those counts? Explain.
47 / 80

Worked answer: separate allocation from computation

An allocation enables computation; it is not the computation itself.

ActionResponsible actorReason
Request / grantApplicationMaster / ResourceManagerThe application requests; the cluster allocator grants.
Launch / monitor locallyNodeManagersEach manages containers on its own machine.
Track the applicationApplicationMasterIt coordinates the application’s tasks and progress.
Calculate countsMapReduce task processesYARN provides resources; tasks execute the logic.
48 / 80

MapReduce

Map records. Group keys. Reduce values.

49 / 80

A key chooses the group; a value supplies the data

For word counts, emit the word as the key and 1 for each occurrence.

Question: count occurrences of each wordOne input linecar bus carThree word occurrencesmapEmit = output a pair( car , 1 )Key: carWhich group?Value: 1What gets added?Same word → same group. Emit a pair for each occurrence.
50 / 80

MapReduce groups records with matching keys

Local records become intermediate pairs, then grouped results.

Input wordsMapShuffle + sortReduce → outputcar(car, 1)bus(bus, 1)car(car, 1)car → [1]bus → [1]car → [1, 1]bus → [1](car, 2)(bus, 1)The two car occurrences stay distinct until their counts are added.
Conceptual process loop51 / 80

An input split is logical work, not a stored block

With text input, each RecordReader delivers complete lines even when a line crosses a split or block boundary.

One text file · byte offsets0c1a2r3␣4t5r6a7i8n9↵10b11u12s13↵Stored blocksBlock A · bytes 0–7Block B · bytes 8–13Split 1 → mapper 1Starts at byte 0; completes the first line.0 → “car train”Reads across the block boundary.Split 2 → mapper 2Starts at byte 8; skips the partial line.10 → “bus”Starts its record at the next full line.
52 / 80

Mappers emit what the next stage needs

The mapper splits each line into words and emits one (word, 1) pair for every occurrence.

Mapper codemap(offset, line): for word in tokenize(line): emit(word, 1)Input: 0 → “car bus car”Emitted pairscar(car, 1)bus(bus, 1)car(car, 1)One input record → three loop iterations → three emitted pairs
53 / 80

A combiner sums locally before records travel

An optional combiner reduces transferred records; the reducer must produce the same answer without it.

Inside each mapperAcross the networkAt the reducerMapper Ared: [1,1]local sum(red,2)(red,2)Mapper Bred: [1]local sum(red,1)(red,1)red: 2 + 1red: 3Same final countFor red: send 2 partial-count records instead of 3 occurrence records.
54 / 80

Combiner correctness: the result must stay the same

A combiner reduces transferred data only when local aggregation preserves the final answer.

CheckRequirementExample
Optional executionCorrect with no combine, one combine or repeated combines.Counts stay correct when partial counts are added.
Order and groupingMerge partial results without changing the answer.For sums: (1 + 1) + 1 = 1 + (1 + 1).
Compatible stateKeep a representation that can be merged again.For an average, carry (sum, count).
55 / 80

Averages of averages can be wrong

Unequal group sizes break a naive average-of-averages.

Local valuesLocal meansCompare the resultsA: [10,20]A: mean = 15B: [90]B: mean = 90(15 + 90) / 2 = 52.5Two means get equal weight.120 / 3 = 40120 = 10 + 20 + 90
Conceptual process loop56 / 80

Keep the sum and count until the final division

Merge the sums and counts. Calculate the average once at the end.

Local (sum, count)Merge both fieldsDivide onceA: (30, 2)B: (90, 1)sum: 30 + 90 = 120count: 2 + 1 = 3Average = 40Merged state: (120, 3)
Conceptual process loop57 / 80

A partitioner routes many keys to each reducer

Equal keys use the same reducer. Different keys may also use that reducer.

Mapper recordsPartition by keyReducer partitionscar, train → R0 · bus → R1Mapper A · car:1Mapper A · bus:1Mapper B · car:1Mapper B · train:1R0 · car → [1]R0 · car → [1,1]R1 · bus → [1]R0 · train → [1]R0 · car → [1,1]R0 · train → [1]R1 · bus → [1]
Conceptual process loop58 / 80

Reducer tasks iterate over grouped keys

One reducer task processes both keys and writes both records to one part-file.

Reducer 0 · grouped inputOne reduce call per keyOne output part-filecar → [1,1]reduce(car): 1 + 1 = 2(car,2)train → [1]reduce(train): 1(train,1)
Conceptual process loop59 / 80

Trace six records through two reducer tasks

Count keys and tasks separately.

5 min · Show grouped input and output for R0 and R1. How many reducer tasks run?
Mapper Ared:1, blue:1, red:1Mapper Bblue:1, green:1, red:1Partition rulered/green → R0 · blue → R1
60 / 80

Two reducer tasks handle three distinct keys

Six emitted records form three key groups; two reducer tasks write three totals into two part-files.

Six mapper emissionsTwo reducer tasksTwo output part-filesReducer 0Reducer 1part-r-00000part-r-00001A · (red,1)A · (blue,1)A · (red,1)B · (blue,1)B · (green,1)B · (red,1)green → [1](green,1)red → [1]red → [1,1]red → [1,1,1](red,3)blue → [1]blue → [1,1](blue,2)
Conceptual process loop61 / 80

MapReduce revision: keep these quantities separate

Trace records first, then count key groups, reducer tasks and output files separately.

ConceptMeaningConsequence
Block / input splitStored bytes / logical input work.A RecordReader handles complete records across boundaries.
Map / combineEmit pairs / optionally aggregate locally.A combiner must not change the final result.
Partition / groupChoose reducer / gather equal keys.Equal keys go together; one reducer can handle many keys.
Task / reduce callOne running reducer / one grouped key.Three keys need not mean three reducer tasks.
Output record / part-fileOne emitted result / one task’s output file.In our standard text-output examples, two reducers write two part-files.
62 / 80

Execution trade-offs

What does MapReduce gain and lose by writing intermediate results to disk?

63 / 80

Separate shuffle storage from output between jobs

Map output uses local storage for shuffle; a later job reads the previous job’s saved output.

Within one jobBetween jobsMap outputBuffered, then spilledlocallyShuffleReducers fetch partitionsReduceRead, merge and calculateJob 1 resultWrite output to HDFSStored datasetSurvives task completionJob 2 inputRead the dataset again
64 / 80

Disk-based execution: benefits and costs

Distinguish temporary shuffle storage from saved job output when explaining the trade-off.

StorageBenefitCost or limitation
Local map outputBuffers and spills support data larger than memory.Disk I/O and network shuffle add work.
Saved output between jobsA later job can reuse the stored dataset.Writing and rereading adds I/O and latency.
Failure distinctionLost map output can be recomputed by rerunning its task.Local shuffle data is not an HDFS-replicated final result.
65 / 80

Hadoop in the cloud

How do Hadoop responsibilities map to a managed cloud cluster?

66 / 80

EMR revision: three node roles

Choose a node type by the capacity it adds and the responsibilities it holds.

Node typeRoleConsequence
PrimaryCoordinates cluster services.Coordination differs from worker computation.
CoreRuns tasks and HDFS DataNodes.Removal affects both compute and stored replicas.
TaskAdds task-processing capacity without HDFS DataNodes.More compute does not add HDFS storage.
67 / 80

Cluster lifetime and data lifetime are different

Save and verify needed results outside the EMR cluster before terminating its compute and HDFS storage.

EMR clusterHDFS input + outputUsable while the cluster existsTermination removes this copy.SaveIndependent storageApproved durable locationExample: Amazon Simple StorageService (S3)Read back the saved output.Verify the independent copy and evidence before terminating the cluster.
68 / 80

Check saved evidence. Terminate the cluster.

Closing a notebook does not terminate its cluster.

INPUTFinished notebookUseful resultCluster still runningOUTPUTConfirmed end stateSaved evidencereadableCluster terminatedClose the lifecycleSave result + checkpointTerminate approvedcluster
69 / 80

Explain storage, scheduling and keyed aggregation

5 min · Input: “car bus car”; one reducer. Explain storage, resource allocation, emitted pairs, groups and final file contents. What must you verify before and after shutdown?

StorageClient metadata lookup→ block readSchedulingYARN resources →application workAggregationMap → partition →group → reduceEvidenceChecked output.Terminated cluster.
70 / 80

Model answer: trace the file to a checked result

Storage supplies bytes, YARN supplies resources, and MapReduce groups and totals the emitted pairs.

Input file: “car bus car”1ReadNameNode → locationsDataNodes → file bytes2RunYARN → task resourcesMap → (car,1), (bus,1),(car,1)3Group + sumcar → [1,1] → 2bus → [1] → 1One reducer → one part-file: bus 1, car 2Read the actual output. Save it independently. Confirm cluster termination.
71 / 80

Practice: apply MapReduce to sales totals

Write your answer before viewing the solution. Values are illustrative.

GivenQuestion
Sales: (North,10), (South,7), (North,5).What key and value should each mapper emit?
Partition rule: North → R0; South → R1.Show grouped input and totals for each reducer.
Optional local aggregation.Can a sum combiner be used? Explain why.
Standard text output; both reducers succeed.How many key groups, reducer tasks and part-files?
72 / 80

Worked answer: sales totals by region

The key changes with the business question; the map–group–reduce reasoning stays the same.

StepResultReason
Map(North,10), (South,7), (North,5)Region defines the group; amount is the value.
Group and reduceR0: North [10,5] → 15; R1: South [7] → 7Add values within each region.
CombineSum partial amounts, if the combiner runs.Regrouping sums preserves each region’s total.
Count2 key groups; 2 reducer tasks; 2 part-filesThe supplied partition rule assigns one group to each task.
73 / 80

Conclusion: Hadoop essentials

Review the whole lecture: the problem, the system, the processing and the result.

74 / 80

1 · Big Data and scaling

Choose a remedy that addresses the actual cause of the problem.

Key pointWhat you should explain
Five VsVolume: amount. Velocity: arrival rate. Variety: formats. Veracity: trust. Value: useful outcome.
Big DataThe data and task strain available capacity; there is no universal size threshold.
Scale up / scale outAdd resources to one host / add hosts and divide the work.
Cost of distributionParallel work adds communication, coordination and partial-failure handling.
Correctness and capacityValidate records to fix wrong results; measure bottlenecks to address slow results.
75 / 80

2 · Hadoop roles and HDFS storage

A replica is another copy of a block, not another unique input record.

Key pointWhat you should explain
Three responsibilitiesHDFS stores files; YARN allocates resources; MapReduce processes records.
Metadata / bytesNameNode tracks files, blocks and locations; DataNodes store block replicas.
Read / writeRead: get locations, then fetch bytes. Write: get placement, then stream a replica pipeline.
Replica failureRead a surviving copy; restore replication. Copies improve availability after node loss.
Workload suitabilityLarge sequential scans fit HDFS. Tiny files add metadata overhead; frequent record updates need a suitable database.
76 / 80

3 · YARN resource management

Keep cluster-wide allocation, application coordination and local task execution separate.

Actor or stepWhat you should explain
ResourceManagerAllocates cluster resources; it does not execute the application’s calculation.
ApplicationMasterCoordinates one application, requests containers and tracks task progress.
NodeManagerLaunches and monitors containers on one worker machine.
ContainerAn allocation of resources such as CPU and memory; not necessarily Docker.
Application lifecycleSubmit → start AM → request task resources → launch and track work → release resources.
77 / 80

4 · MapReduce: input to output

Example: car, bus, car → (car,1), (bus,1), (car,1) → car:[1,1], bus:[1] → car:2, bus:1.

Stage or distinctionWhat you should explain
Block / split / recordBlock: stored bytes. Split: logical mapper input. RecordReader delivers records.
MapEmit key–value pairs. For word count: each occurrence emits (word, 1).
Partition and shuffleRoute equal keys to the same reducer; transfer, sort and group intermediate pairs.
ReduceProcess each grouped key and its values to emit final results.
Keys / tasks / filesOne reducer can process several keys. In our standard text-output examples, each reducer writes one part-file.
78 / 80

5 · Combiners and execution trade-offs

A performance optimisation is valid only if it preserves the required result.

Key pointWhat you should explain
Optional combinerLocally aggregate to reduce transfer; the answer must survive zero, one or repeated combines.
Safe partial resultsMerging partial state must preserve the answer under regrouping and reordered inputs.
Average pitfall[10,20] and [90]: mean of means = 52.5, but the true mean is 120 / 3 = 40.
Correct average stateCarry (sum, count): (30,2) + (90,1) = (120,3). Divide once at the end.
Disk distinctionShuffle uses local map output; saved output can feed later jobs. Disk and network I/O add cost.
79 / 80

6 · EMR and a complete Hadoop answer

You should be able to explain the system and work through a new example without copying a diagram.

Key pointWhat you should explain
Primary / core / taskPrimary coordinates. Core provides compute and HDFS storage. Task adds compute without HDFS DataNodes.
Cluster lifetimeCluster HDFS is tied to the cluster; preserve required results outside it before termination.
Verified resultRead the saved output, check its meaning and verify the independent copy.
Finish the labSave required evidence and terminate the cluster. Closing a notebook does not stop the cluster.
Answer clearlyState the rule → trace the mechanism → show the result → justify the choice or trade-off.
80 / 80

Spark architecture
and pipelines

Architecture / DataFrames / Execution

From a query to distributed computation.

INF2006 · CLOUD COMPUTING AND BIG DATA001

One pipeline, two views: code and execution

Predict when Spark runs the work and where it stores each result.

Expressfilter → group → sumPlanChoose physical workExecuteTasks on partitionsVerifyN:15 · S:20
INF2006 · BIG DATA 002

Our question: how much did each region sell?

Two North sales add to 15; the single South sale totals 20.

Three salesGroup by region, then addRegional totalsRegionAmountNorth10South20North5NorthSouth1010 + 520North15South20Each row is one sale.
INF2006 · BIG DATA 003

Spark execution

How does Spark organise several operations into one execution graph?

INF2006 · BIG DATA 004

Spark uses a graph to plan multiple operations

Flexible APIs and data reuse can reduce work. Measure the execution time.

MapReduceStore one job’s outputRead it in the next jobSparkPlan connected operationsReuse data when persisted
INF2006 · BIG DATA 005

MapReduce materialises boundaries between jobs

Intermediate storage provides boundaries and also costs I/O.

Job 1 · filterStored resultJob 2 · sumRegionAmountNorth10South20North5Intermediate filesNorth 10South 20North 5Write → next job readsRegionTotalNorth15South20
INF2006 · BIG DATA 006

Spark can plan multiple operations together

A shared engine can optimise a connected pipeline.

FilterGroup by regionSumRegionAmountNorth10South20North5RegionAmountsNorth10, 5South20One connected planGrouping still needs an exchange.RegionTotalNorth15South20
INF2006 · BIG DATA 007

Spark still requires processing and data transfer

A different API does not remove hardware limits.

CPUParse and calculateWork per recordNetworkMove shuffle recordsWork between executorsDisk / memoryRead, spill and storeCapacity still matters
INF2006 · BIG DATA 008

Spark architecture

Which work belongs to the driver, and which work belongs to executors?

INF2006 · BIG DATA 009

The driver assigns work. Executors run it.

Each task processes one partition on an executor.

DriverPlans and assigns tasksFilter partition 0Filter partition 1Executor AFilter partition 0Executor BFilter partition 1Filter partition 0Filter partition 1
INF2006 · BIG DATA 010

Each partition gets its own task

Within this stage, two partitions produce two tasks—even though they contain three rows.

Input partitionsFilter stageOutput partitionsPartition 0North10South20Task 0Keep North rowsPartition 0North10Partition 1North5Task 1Keep North rowsPartition 1North5
INF2006 · BIG DATA 011

Four partitions can run in two waves

With two available task slots, four equal-length tasks need two waves.

Wave 1Wave 2Time →Slot 1Task · partition 0Task · partition 2Slot 2Task · partition 1Task · partition 3Illustration: equal task durations.
INF2006 · BIG DATA 012

Assign responsibilities before drawing arrows

5 min · Who filters input partitions? Who schedules work? Where does collect() send the final rows?

DriverWho schedules tasks?Where does collect end?ExecutorsWho filters records?Who writes part-files?Cluster managerWho allocates resources?Example: YARN
INF2006 · BIG DATA 013

Plan on the driver; process on the executors

The cluster manager supplies resources; collect() returns rows to the driver.

DriverExecutorSchedules the taskRuns the filterKeep North rowsKeep North rowsNorth | 10North | 10collect() resultThe cluster manager supplies executor resources.
INF2006 · BIG DATA 014

Data models

How do RDDs and DataFrames represent distributed data?

INF2006 · BIG DATA 015

Two ways to represent distributed data

Structured information gives Spark SQL more options when it plans work.

Resilient Distributed DatasetRDD: objects in partitions("N", 10)DataFrameNamed, typed columnsregion: STRING · amount: DOUBLE
INF2006 · BIG DATA 016

An RDD represents records as objects

Object-level transformations make the logic explicit.

Pairs in an RDDAdd amounts for each regionResult("North", 10)("South", 20)("North", 5)pairs.reduceByKey( lambda a, b: a + b)("North", 15)("South", 20)North: 10 + 5 = 15South: 20
INF2006 · BIG DATA 017

A DataFrame has named columns with data types

A schema gives the engine information for planning.

Rows in a DataFrameAdd amounts for each regionResultregion: stringamount: doubleNorth10South20North5(sales.groupBy("region") .sum("amount"))regionsum(amount)North15South20North: 10 + 5 = 15South: 20
INF2006 · BIG DATA 018

Use a schema when the question is relational

Choose an abstraction for the work. Measure its performance.

Named columns and relationships(sales.groupBy("region") .sum("amount"))DataFrame / SQLStart here for filters, joins and totals.Custom operations on objectsobjects.map(custom_function)RDDUse when lower-level control is needed.
INF2006 · BIG DATA 019

A schema defines how to interpret values

The schema assigns column names and types; validation checks the values.

CSV textNorth,42.5,2026-01-01regionSTRINGNorthamountDOUBLE42.5dateDATE2026-01-01
INF2006 · BIG DATA 020

Data ingestion

How do we turn source files into rows with predictable types?

INF2006 · BIG DATA 021

Explicit schemas make a CSV read predictable

An explicit schema names the fields and tells Spark how to parse them.

Read with named, typed columnsschema = "region STRING, amount DOUBLE"sales = (spark.read .schema(schema) .option("header", True) .csv("sales.csv"))sales.printSchema()printSchema()root |-- region: string |-- amount: doubleChecks column structure.Does not validate every value.
INF2006 · BIG DATA 022

A declared type does not guarantee a valid value

Inspect failed parses and missing values before filtering or aggregating.

CSV inputParse amount as DOUBLETyped resultregionamountNorth"10"South"oops""10" → 10.0"oops" is not numericFailed parse → nullregionamountNorth10.0Southnullnull means missing or unknown—not zero.
INF2006 · BIG DATA 023

Programming model

Which operations build a plan, and which operations execute it?

INF2006 · BIG DATA 024

Transformations build a new plan

select, filter, limit and aggregation definitions describe work.

Your codevalid = sales.filter("amount > 0")selected = valid.select( "region", "amount")Plan for selectedRead salesKeep amount > 0Select region, amountNo result rows requested yet.
INF2006 · BIG DATA 025

Actions request a result or an output

show, count, collect and writes have different destinations.

show / countBounded display / one numberReturn a small answercollectEvery result row → driverResult must fit driver memorywritePartitions → shared storageKeep large output distributed
INF2006 · BIG DATA 026

show and count return different evidence

A sample shows values. A count shows the number of rows.

Five distributed rowsPartitionRows0North 100South 200North 51South 71North 8sales.show(2)sales.count()RegionAmountNorth10South20Up to two displayed rowsIllustrative order; no sort applied.5Total number of rows
INF2006 · BIG DATA 027

collect returns every result row to the driver

Limit the result size before you use collect for display.

Executor resultscollect()Driver listExecutor 1North | 15North | 15Executor 2South | 20South | 20North | 15South | 20
INF2006 · BIG DATA 028

A distributed write keeps the result distributed

A path can describe a dataset containing several files.

Executor resultswrite.parquet()Shared storageExecutor 1North | 15part-00000North | 15Executor 2South | 20part-00001South | 20North | 15South | 20
INF2006 · BIG DATA 029

Predict what limit(5) does before show()

Discuss your prediction; the next slide reveals the answer.

Read these two linesb = sales.filter("amount > 0").limit(5)b.show()Predict before running1. What does b contain?2. Which line requests rows?3. Must five rows appear?
INF2006 · BIG DATA 030

limit(5) changes the plan; show() requests rows

The variable b is a DataFrame, not a Python list containing five rows.

Read these two linesb = sales.filter("amount > 0").limit(5)b.show()What happensb is a DataFrame plan.show() requests work.At most five rows are displayed.Without orderBy(), which five rows is not guaranteed.
INF2006 · BIG DATA 031

Execution dependencies

How does Spark turn a chain of transformations into jobs, stages and tasks?

INF2006 · BIG DATA 032

Dependencies form a directed acyclic graph

Read this graph as data dependencies, not a timeline of machines.

Directed acyclic graph (DAG)Read salesKeep positive salesGroup + sumNeeds the read resultNeeds the filtered rows
INF2006 · BIG DATA 033

A shuffle separates dependent stages

Spark can combine local operations in a pipeline. Grouping may redistribute rows.

Stage 1 · local workStage 2 · mergeShufflePartition 0Read → filter → partial sumN:10 · S:20Partition 1Read → filter → partial sumN:5North: 10 + 5 = 15South: 20
INF2006 · BIG DATA 034

Matching region keys must meet to form the total

This example routes partial sums by key, then merges each region’s values.

Local partial sumsGroup matching regionsMergePartitionRegionSum0North100South201North5North10 + 515South202010
INF2006 · BIG DATA 035

Application, jobs, stages and tasks describe different scales

Use the Spark UI to test the explanation against a real run.

ApplicationDriver + executorsJobWork for a computationStageA set of partition tasksTask · P0Task · P1Task · P2One illustrated branch of the hierarchy.
INF2006 · BIG DATA 036

Query planning

How can Spark change an execution plan while preserving the result?

INF2006 · BIG DATA 037

The optimizer rewrites valid relational work

It searches for a better plan. It cannot correct business logic.

Written expressionEligible rewritten planRead region, amount, commentSelect region, amountFilter amount > 0Read required columnsFilter amount > 0Return region, amountSame result; scan savings depend on the data source.
INF2006 · BIG DATA 038

explain() reveals the plan before a large run

Find scan columns, filters and exchanges. Compare the plans. Measure performance.

Inspect the plantotals = (sales .filter("amount > 0") .groupBy("region") .sum("amount"))totals.explain("formatted")Illustrative physical planScanRead required columnsFilterKeep positive amountsPartial aggregateLocal sumsExchangeRedistribute by regionFinal aggregateMerge sums
INF2006 · BIG DATA 039

Spark SQL

How can SQL and DataFrame code express the same computation?

INF2006 · BIG DATA 040

SQL and DataFrame APIs can express the same query

A temporary view gives a DataFrame a name for queries in this session.

DataFrame APISQL(sales.filter("amount > 0") .select("region", "amount"))SELECT region, amountFROM salesWHERE amount > 0Register first: sales.createOrReplaceTempView("sales")Both express the same positive-sales query.
INF2006 · BIG DATA 041

Word count

How does a familiar MapReduce example translate into Spark?

INF2006 · BIG DATA 042

Word count still groups keys, with fewer API steps

The same word-count logic: tokenize the text, group equal words, then count.

Text inputSplit + explodeGroup + count"car bus car"carbuscarwordcountcar2bus1read.text(path)split → explodegroupBy("word").count()

Relational operations

How do joins and aggregations change the shape and number of rows?

INF2006 · BIG DATA 044

A unique lookup adds attributes to each sale

A unique matching lookup adds attributes without multiplying a sale.

SalesLookup matchesJoined rowsSale · customer · amountCustomer · regionSale · customer · amount · regions1 | C1 | 10s2 | C2 | 20C1 | NorthC2 | Souths1 | C1 | 10 | Norths2 | C2 | 20 | South2 rows · total 30
INF2006 · BIG DATA 045

Before a join, define what each left row represents

A row’s meaning is part of the join specification.

Three distinct transactionsSale IDCustomerAmounts1C110s2C15s3C38Repeated customer ≠ duplicate sales1 and s2 are different sales.Both belong to customer C1.Define the row’s meaningbefore checking duplicates.
INF2006 · BIG DATA 046

Duplicate lookup keys can multiply input rows

A lookup needs an explicit uniqueness or multiplicity rule.

SalesLookup matchesJoined rowsSale · customer · amountCustomer · regionSale · customer · amount · regions1 | C1 | 10C1 | NorthC1 | Norths1 | C1 | 10 | Norths1 | C1 | 10 | North2 rows · total 20
INF2006 · BIG DATA 047

An inner join can duplicate and discard rows

An inner join keeps matches, multiplies repeated matches, and drops unmatched sales.

SalesLookup matchesJoined rowsSale · customer · amountCustomer · regionSale · customer · amount · regions1 | C1 | 10s2 | C1 | 5s3 | C3 | 8C1 | NorthC1 | Norths1 | C1 | 10 | Norths1 | C1 | 10 | Norths2 | C1 | 5 | Norths2 | C1 | 5 | North4 rows · total 30C3: no match → no output row
INF2006 · BIG DATA 048

Predict the result before running the pipeline

5 min · Trace the amount > 0 filter. Join by customer ID. Sum by region. What if C1 occurs twice in the lookup?

SalesCustomerAmountC110C1−2C220Unique customer lookupCustomerRegionC1NorthC2SouthPredict the totals1. Keep amount > 02. Join on customer3. Sum by region
INF2006 · BIG DATA 049

Two positive sales become two regional totals

With unique lookup keys: North = 10 and South = 20.

SalesC1 | 10C1 | −2C2 | 20Filter > 0C1 | 10C2 | 20Join regionNorth | 10South | 20Sum by regionNorth | 10South | 20Remove the negative saleNorth 10 · South 20
INF2006 · BIG DATA 050

Reliability

How can Spark recover a lost partition from the operations that produced it?

INF2006 · BIG DATA 051

Spark can calculate lost partitions again from lineage

Lineage can rebuild lost computed data when its inputs remain available.

Available source + lineageLost executorReplacement executorPartition 1: North 5Keep North rowsNorth 5Computed partition lostRun filter againNorth 5
INF2006 · BIG DATA 052

Pipeline output

How do we store distributed results without collecting all rows on the driver?

INF2006 · BIG DATA 053

Write distributed results before you collect rows for display

Use a bounded aggregate for notebook visualization.

Write, then read the datasetpath = "output/region_totals"totals.write.parquet(path)check = spark.read.parquet(path)check.printSchema()check.show(5)One dataset directorypart-00000…part-00001…Read the directory path.
INF2006 · BIG DATA 054

Read the saved dataset back and check its meaning

Verify schema, row count and totals at the output path—not only notebook success.

Saved Parquet datasetregiontotalNorth15South20Verify what you read backColumn typesregion: string · total: numericResult rows2 regionsBusiness total15 + 20 = 35
INF2006 · BIG DATA 055

Spark lab

How does a Zeppelin notebook submit work to the Spark cluster?

INF2006 · BIG DATA 056

Zeppelin sends work to the Spark cluster

A browser interface, a driver and executors have distinct roles.

Zeppelin notebookSpark driverExecutorscounts.show(5)Plan + scheduleCount wordsHDFS datacar: 2 · bus: 1car: 2 · bus: 1The browser displays results; executors process the data.

Cluster lifecycle

What must happen after the notebook finishes?

INF2006 · BIG DATA 058

A completed notebook does not terminate its cluster

Save evidence. Close connections. Check cluster termination.

NotebookCluster stateOutput savedBrowser closedAfter closing the browserStill runningAfter explicit terminationTerminatedExplicitly terminate → verify the cluster state.

Use a typed, traceable pipeline in Lab08

5 min · Explain when a census aggregation runs. Identify where tasks execute. State the checks before plotting and cluster termination.

Know the dataTypes · missing valuesOne row means what?Trace the workPlan → action → tasksWhere must data move?Verify the resultCounts · joins · totalsSave, then terminate

Spark performance
and storage

Shuffle / Storage / Evidence

Read the plan. Measure data movement.

INF2006 · CLOUD COMPUTING AND BIG DATA001

A correct query can still use many resources

Find the resource cost before you change a setting.

ReadBytes and filesMoveShuffle acrossworkersComputeCPU and memoryWriteDurable outputinputrecordsdistributedresultpersistresult
INF2006 · BIG DATA 002

Query planning

Where does the execution plan reveal the cost of a query?

INF2006 · BIG DATA 003

The driver chooses a physical plan

The optimizer uses rules and available information. It does not guarantee the fastest execution.

Logical intentColumns, predicates, joinsOptimized intentEquivalent rewritesPhysical operatorsScan, exchange, aggregaterewriteequivalentquerychooseoperators
INF2006 · BIG DATA 004

Read the plan before guessing

Find scans, filters and exchanges in an explain output.

from pyspark.sql import functions as F
q = df.filter(F.col("age") >= 30)
q = q.groupBy("occupation").count()
q.explain(mode="formatted")
ScanRequired columnsFilterRemaining predicateExchangeShuffle boundaryread columnsroute matching records
INF2006 · BIG DATA 005

Read less data

How can column selection and filtering reduce the work?

INF2006 · BIG DATA 006

Select the columns the question needs

A source with many columns does not require all columns in the working dataset.

small = df.select(
    "age", "occupation", "hours_per_week"
)
small.filter("age >= 30").show(5)
15 columnsAdult census source3 columnsage, occupation, hoursSmaller inputLess downstream workproject requiredcolumnscarry smaller records
INF2006 · BIG DATA 007

A filter can move toward the source

Predicate pushdown and row filtering are related but different.

Query filterage >= 30Reader predicateSupported pushdownResidual filterKeep exact matching rowspasssupportedpredicateverifysurvivingrows
INF2006 · BIG DATA 008

Predict which data the query can skip

A partition filter and a value filter may skip different work.

5 min · Which work can be skipped by dt and age filters?
Directorydt=2026-09-30Row groupage range 18–25Predicatedt matches · age >= 30
INF2006 · BIG DATA 009

Data movement

Which operations keep data local, and which move it between workers?

INF2006 · BIG DATA 010

Narrow work stays within an input partition

Filter and projection usually avoid redistributing records.

Partition ARows 1–100Local filterKeep matching rowsOutput ASubset of Aevaluatelocallyemitsubset
INF2006 · BIG DATA 011

Grouping combines records with equal keys

A shuffle redistributes records so tasks can process each complete group.

Input partition 1Input partition 2Key group AKey group BGroup by key · illustrative recordsA1A1A1B1B1B1A2A2A2B2B2B2BeforeAfter · keys colocated conceptually
INF2006 · BIG DATA 012

Exchange separates stages

Tasks in one stage process partitions. The next stage reads shuffled output.

Stage 1Read and partial aggregateShuffle filesIntermediate blocksStage 2Final aggregatewriteintermediateblocksfetchshuffledblocks
INF2006 · BIG DATA 013

Parallelism

How do partition count and partition size affect the work?

INF2006 · BIG DATA 014

Partitions divide work, not ownership

Too few or too many tasks can reduce performance.

Too fewIdle capacity · large tasksBalancedUseful task sizesToo manyScheduling and tiny outputs
INF2006 · BIG DATA 015

Too few partitions can leave capacity idle

Available workers need independent runnable tasks.

INPUTInput layoutInput: one partitionSeveral availableslotsOUTPUTConsequencesLimited parallel workOne large taskSchedule one taskOne slot worksOther slots wait
INF2006 · BIG DATA 016

Partitions let tasks process limited amounts of data concurrently

Balance task size against scheduling overhead.

INPUTInput layoutP1: bounded rowsP2: bounded rowsP3: bounded rowsOUTPUTConsequencesSeveral partialresultsMeasured final costSchedule task slotsTasks can run togetherEach fits availablememory
INF2006 · BIG DATA 017

Too many tiny partitions multiply overhead

More partitions can increase overhead after parallelism is sufficient.

INPUTInput layoutMany tiny inputsLittle work per taskOUTPUTConsequencesScheduling costSmall-file pressureSchedule repeatedlyLaunch many tasksHandle many small outputs
INF2006 · BIG DATA 018

Repartition redistributes data through an exchange

Use it when the benefit of a new layout exceeds the exchange cost.

balanced = df.repartition(8, "occupation")
summary = balanced.groupBy("occupation").count()
summary.explain("formatted")
Existing layoutUneven or unsuitableShuffleRedistribute by keyNew partitionsDownstream workloadredistributerecordsemit newlayout
INF2006 · BIG DATA 019

Coalesce can reduce tasks without a full shuffle

A drastic reduction can also reduce parallelism.

5 min · A team uses coalesce(1) to make a single large output file. Predict the trade-off.
Many partitionsSmall output fragmentsFewer partitionsCombine existing workTrade-offPotential imbalance
INF2006 · BIG DATA 020

Many partitions can become many output fragments

Inspect output sizes before changing partition counts.

INPUTUpstream tasksP1: few rowsP2: few rowsP3: few rowsOUTPUTFragmented directorySmall part-filesMore file-open workWrite current layoutSeparate task outputsRecord actual file sizes
INF2006 · BIG DATA 021

Coalesce combines existing task partitions

A narrow merge can reduce tasks without evenly redistributing rows.

INPUTExisting partitionsP1: rows AP2: rows BP3: rows COUTPUTFewer tasksTask X: A + BTask Y: Ccoalesce(2)Use fewer downstreamtasksNo full rebalance bydefault
INF2006 · BIG DATA 022

One output task can become the bottleneck

Writing one output file may reduce throughput.

INPUTRequested outputAll result rowsSeveral compute slotsOUTPUTTrade-offFewer filesLess parallelismPotential memorypressurecoalesce(1)One final taskOne task owns the output
INF2006 · BIG DATA 023

One frequent key can delay the entire stage

The stage finishes when its slowest required task finishes.

5 min · Why might doubling workers barely help a stage with one oversized key?
Small tasksMost keys finishHot keyMany unknown occupationsLong tailOne task still running
INF2006 · BIG DATA 024

Join strategies

How does the join strategy change the data sent between workers?

INF2006 · BIG DATA 025

The join strategy changes data movement

Large–large joins and small lookup joins have different costs.

Shuffle joinRedistribute matching keysBroadcast joinReplicate a small sideDecisionSize, join type, memory
INF2006 · BIG DATA 026

A shuffle join redistributes matching keys

Large inputs can require significant exchange work.

INPUTDistributed factsA: C1, C2B: C1, C3OUTPUTMatched resultsC1 pairs colocatedOther keys processedExchange by join keyMatching keys meetJoin matching groups
INF2006 · BIG DATA 027

A broadcast join replicates a small lookup

Replication uses network capacity and memory so workers can match records locally.

INPUTLookup + factsLookup: C1 → NorthFacts remainpartitionedOUTPUTLocal matchesFact task A matches C1Fact task B matches C1Distribute the lookupCopy lookup to workersJoin each fact partition
INF2006 · BIG DATA 028

Choose a join strategy using evidence

Measure correctness and cost on equivalent work.

INPUTEvidence to inspectLookup sizeKey skewJoin typeOUTPUTControlled comparisonEquivalent outputSame input + clusterRepeated timingsInspect physical planStrategy selectedExchange / broadcast
INF2006 · BIG DATA 029

Broadcast a measured small lookup

Check the keys. Inspect the physical plan.

from pyspark.sql.functions import broadcast
joined = df.join(
    broadcast(occupation_lookup), "occupation", "left"
)
joined.explain("formatted")
Fact rowsStay partitionedLookupSent to workersMatchLocal join where supportedfact recordsreplicatedlookup
INF2006 · BIG DATA 030

Caching

When does reusing a stored result avoid enough work to justify caching?

INF2006 · BIG DATA 031

Cache data that repeated operations reuse

Caching is optional. Cached data uses memory that execution also needs.

Clean onceShared expensive prefixCacheReusable partitionsTwo consumersAnalysis and exportmaterializeonactionreuse ifavailable
INF2006 · BIG DATA 032

An action stores the data after cache registration

Release cached data after the repeated work.

clean = df.filter("age >= 18").cache()
clean.count()  # materialize for this example
clean.groupBy("occupation").count().show()
clean.groupBy("income").count().show()
clean.unpersist()
cache()Request persistencecount()Materialize neededpartitionsunpersist()Release persistenceactionloadspartitionsreleaseafterreuse
INF2006 · BIG DATA 033

Memory pressure

What can disk spill tell us about the workload?

INF2006 · BIG DATA 034

Disk spill is a symptom to investigate

Execution memory, cached data and task size interact.

Task working setSort or aggregate buffersMemory pressureNot everything fitsSpill filesDisk I/O and later mergeworkingsetexceedscapacitywritetemporarybuffers
INF2006 · BIG DATA 035

Storage

How do file format and layout affect the amount of data a query reads?

INF2006 · BIG DATA 036

Parquet stores columns together

Column selection and compression can reduce analytical I/O.

CSV rowsText parsing and delimitersParquet columnsTyped column chunksRead querySelect relevant columns
INF2006 · BIG DATA 037

A CSV reader interprets row-oriented text

Projection does not make CSV a columnar file.

INPUTCSV linesN,10,2026-10-01S,20,2026-10-02OUTPUTTyped rowsregion STRINGamount DOUBLEdt DATEParse with a schemaSplit encoded fieldsCast each declared type
INF2006 · BIG DATA 038

Parquet groups typed values by column

Column projection can reduce read work for supported queries.

INPUTColumn chunksregion: N, Samount: 10, 20dt: Oct01, Oct02OUTPUTProjected rowsN | 10S | 20Read selected columnsNeed region + amountSkip unneeded dt chunk
INF2006 · BIG DATA 039

Statistics can exclude a row group that cannot match

Statistics can exclude data. They do not replace correct row filtering.

INPUTRow-group evidenceRG1 age: min18 max25RG2 age: min30 max60OUTPUTRead + filterRG1 excluded ifsupportedRG2 rows still testedPredicate: age ≥ 30RG1 cannot matchRG2 may match
INF2006 · BIG DATA 040

A Spark write creates a dataset directory

Read the saved dataset. Check its schema and content.

path = "/user/hadoop/results/occupation_hours"
summary.write.mode("overwrite").parquet(path)
check = spark.read.parquet(path)
check.printSchema()
check.show(5)
DataFrameDistributed partitionsDirectoryPart files and metadataReadbackTypes and valueswrite partfilesreaddatasetback

Partition directories target repeated filters

Choose useful keys without creating millions of tiny files.

path = "/tmp/teaching_events"
events.write.mode("overwrite") \
    .partitionBy("dt").parquet(path)
recent = spark.read.parquet(path)
recent.filter("dt = '2026-10-01'").explain()
dt=2026-09-30One directory partitiondt=2026-10-01Another directory partitionFiltered queryRead relevant datesread matching datedirectory
INF2006 · BIG DATA 042

Measurement

How do we compare performance using equivalent work and controlled conditions?

INF2006 · BIG DATA 043

Time equivalent work under controlled conditions

Plan construction time is not query execution time.

6 min · Design a fair CSV-versus-Parquet comparison. What must stay fixed?
Same questionSame rows and resultSame environmentVersion, resources, settingsRepeated actionsCold and warm separately
INF2006 · BIG DATA 044

A useful result explains both answer and cost

Numbers need the exact query and environment.

01CorrectnessCounts and groups agree02Cost evidenceScan, shuffle, spill, tasks03ContextVersion and configuration
INF2006 · BIG DATA 045

Pipeline lab preparation

Which data and format assumptions must we check before running the lab?

INF2006 · BIG DATA 046

Read the lab data description critically

Lab08 downloads Adult census data. The original description differs.

Downloaded fileadult.data → adult.csvActual schema15 census fieldsMissing categoryInspect trimmed values

Improve one measured bottleneck at a time

Correctness comes before a performance claim.

01InspectPlan and task metrics02ChangeOne justified variable03VerifySame result. Measured cost.
INF2006 · BIG DATA 048

Spark ML and
valid evaluation

Features / Training / Evaluation

Build the pipeline. Check the evaluation.

INF2006 · CLOUD COMPUTING AND BIG DATA001

A pipeline becomes a predictive model

Define the question before selecting an algorithm.

Past recordsKnown outcomesLearningPatterns in featuresNew recordsPredicted outcomesfit onpastexamplesapplylearnedmodel
INF2006 · BIG DATA 002

Different questions need different targets

Classification, regression and clustering solve different problems.

ClassificationChurn: yes or noRegressionSpending next monthClusteringSimilar customer groups
INF2006 · BIG DATA 003

Classification predicts a category

Probability and the chosen class are different outputs.

INPUTExamplesCustomer featuresKnown past churnlabelsOUTPUTOutputsP(churn) = 0.7Threshold → class1Fit a classifierLearn from training rowsScore unseen features
INF2006 · BIG DATA 004

Regression predicts a numeric quantity

Evaluate errors in units meaningful to the decision.

INPUTExamplesHouse featuresKnown training pricesOUTPUTOutputsPredicted price: 240Observed price: 250Residual: 10Fit a regression modelLearn numericrelationshipPredict unseen features
INF2006 · BIG DATA 005

Clustering groups examples without target labels

A discovered group is not automatically a business label.

INPUTExamples[1,1], [1,2][8,8], [8,9]OUTPUTOutputsGroup A: near [1,1.5]Group B: near [8,8.5]Group by representationFind nearby featurevectorsChoose useful distance
INF2006 · BIG DATA 006

Data contract

What does one row represent, and what outcome will the model predict?

INF2006 · BIG DATA 007

One row must match the prediction unit

Features must exist at the time of the decision.

UnitCustomer at snapshot dateFeaturesKnown before predictionLabelOutcome in future windowcaptureas-ofvaluesobservelateroutcome
INF2006 · BIG DATA 008

Evaluation design

How do we keep information from the evaluation data out of training?

INF2006 · BIG DATA 009

Split before learning preparation parameters

Do not use test data to fit pipeline stages.

Raw rowsCommon fixed cleaningTraining rowsFit learned stagesHeld-out rowsTransform with fitted stagessplithold asideapplyfittedparameters
INF2006 · BIG DATA 010

Random splits suit exchangeable rows

A seed helps repeat an experiment, but row identity also matters.

train, valid, test = raw.randomSplit(
    [0.6, 0.2, 0.2], seed=42
)
# Keep the same input and saved split identities.
TrainingLearn parametersValidationChoose settingsTestFinal estimate
INF2006 · BIG DATA 011

Use time boundaries for future predictions

Future records must not train a model evaluated in the past.

5 min · A January model uses a February-updated account status. Why is its January test invalid?
01EarlierTrain02LaterValidate and select03LatestFinal holdout
INF2006 · BIG DATA 012

Leakage often looks like a helpful feature

A perfect score can indicate an invalid experiment.

5 min · Which leak does a scaler fitted on the entire dataset create?
Direct leakClosed-account reasonPreparation leakScale using test rowsEntity leakSame customer on both sides
INF2006 · BIG DATA 013

Direct leakage exposes an outcome to the model

A feature must exist at the prediction time.

INPUTPrediction at MondayAccount ageMonday usageFriday closed_reasonOUTPUTValid inputAge + usageNo Friday reasonConstruct featuresExclude later outcomeUse Monday-known data
INF2006 · BIG DATA 014

Preparation leakage lets held-out rows influence fit

Learned preprocessing belongs inside the training boundary.

INPUTBefore splittingTraining rowsHeld-out rowsOUTPUTCorrect isolationFit scaler on trainingTransform held-outonlyWrong: fit scaler onbothMean includes held-outdataThen evaluate held-outrows
INF2006 · BIG DATA 015

Entity leakage repeats a person across the split

The split should match who or when the model must generalise to.

INPUTRepeated observationsU1: JanuaryU1: FebruaryU2: JanuaryOUTPUTEvaluation boundaryUnseen users: groupsplitFuture rows: timesplitSplit by intended taskRandom rows can share U1Hold out users ifrequired
INF2006 · BIG DATA 016

Spark ML

How does Spark carry features and outcomes through a learning workflow?

INF2006 · BIG DATA 017

DataFrames carry both features and outcomes

spark.ml is the DataFrame-based pipeline API.

Input columnsStrings and numeric fieldsfeaturesOne vector columnlabelKnown numeric targetencodeandassemblepredictors
INF2006 · BIG DATA 018

Estimators fit. Transformers transform.

Fitting an Estimator returns a fitted Model.

5 min · Classify StringIndexer, VectorAssembler and a trained LogisticRegressionModel. Which calls learn?
Estimatorfit(training DataFrame)ModelLearned parametersTransformertransform(DataFrame)returnsfittedmodelapplieslearnedparameters
INF2006 · BIG DATA 019

Features and labels

How do we convert source columns into the inputs and outcomes a model needs?

INF2006 · BIG DATA 020

Declare the positive class explicitly

Set churn yes = 1 to avoid labels that depend on category frequency.

from pyspark.sql import functions as F
known = raw.filter(F.col("Churn").isin("Yes", "No"))
labeled = known.withColumn(
    "label", (F.col("Churn") == "Yes").cast("double")
)
No churnlabel=0.0Churnlabel=1.0InterpretationScore targets class1
INF2006 · BIG DATA 021

Category numbers are identifiers

StringIndexer learns a category mapping from training data.

from pyspark.ml.feature import StringIndexer
indexer = StringIndexer(
    inputCol="Contract", outputCol="contract_idx",
    handleInvalid="keep"
)
index_model = indexer.fit(train)
encoded = index_model.transform(valid)
Contract stringsMonthly, annualFitted vocabularyCategory → indexNew categoryUse a declared policylearnmappingapplyunknownpolicy
INF2006 · BIG DATA 022

Encode categories without an implied order

One-hot vectors separate category membership.

from pyspark.ml.feature import OneHotEncoder
encoder = OneHotEncoder(
    inputCols=["contract_idx"],
    outputCols=["contract_vec"]
)
Index0,1,2VectorCategory indicatorsModel inputNo implied rankencode categorymembershipassemble modelrepresentation
INF2006 · BIG DATA 023

Cleaning changes which records train the model

Count exclusions and inspect types before assembly.

typed = raw.withColumn(
    "TotalCharges", F.col("TotalCharges").cast("double")
)
usable = typed.filter(F.col("TotalCharges").isNotNull())
# Audit how many records this policy removes.
Raw TotalChargesNumeric strings · blanksCastNumbers or nullsPolicyExclude or train-fitimputationparsenumericvaluesapplyexclusionpolicy
INF2006 · BIG DATA 024

The assembler fixes feature order

A vector must mean the same thing at training and prediction.

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(
    inputCols=["tenure", "MonthlyCharges", "contract_vec"],
    outputCol="features"
)
tenure36MonthlyCharges70features[36,70,…]vector position0vectorposition1
INF2006 · BIG DATA 025

Scaling learns statistics from training

Centering sparse vectors can increase memory use.

from pyspark.ml.feature import StandardScaler
scaler = StandardScaler(
    inputCol="features", outputCol="scaledFeatures",
    withStd=True, withMean=False
)
Training distributionMeans and variancesScalerModelFrozen parametersValidation rowsSame transformationfitstatisticstransformusingfrozenvalues
INF2006 · BIG DATA 026

Convert text to a numeric representation

Tokenize the text. Create a numeric vector.

Text"service stopped today"Tokensservice / stopped / todayVectorTerm counts or hashestokenizevectorizetokens
INF2006 · BIG DATA 027

ML pipelines

How do we keep preparation and learning steps in the correct order?

INF2006 · BIG DATA 028

A pipeline packages ordered stages

Fit the entire learned workflow on training rows.

train, valid, test = labeled_clean.randomSplit(
    [0.6, 0.2, 0.2], seed=42
)
lr = LogisticRegression(labelCol="label")
pipeline = Pipeline(
    stages=[indexer, encoder, assembler, lr])
model = pipeline.fit(train)
predictions = model.transform(valid)
Index + encodeCategorical preparationAssembleNumeric vectorLogistic regressionFit classifierencodethenassemblefit usingfeatures
INF2006 · BIG DATA 029

Training

What does logistic regression learn from the prepared training rows?

INF2006 · BIG DATA 030

Logistic regression predicts a class probability

Learn coefficients rather than memorizing labels.

FeaturesWeighted inputsScoreLearned linear combinationProbabilityLogistic mappingweightedcombinationmapscore toprobability
INF2006 · BIG DATA 031

Prediction

What is the difference between a predicted class and its probability?

INF2006 · BIG DATA 032

Prediction and probability answer different questions

A threshold changes labels without retraining the score model.

predictions.select(
    "label", "rawPrediction", "probability", "prediction"
).show(5, truncate=False)
rawPredictionModel scoreprobabilityClass probabilitiespredictionThresholded decisionscore-to-probabilitymappingconfiguredthreshold
INF2006 · BIG DATA 033

Evaluation metrics

What does each metric tell us about the model's predictions?

INF2006 · BIG DATA 034

ROC AUC measures ranking, not accuracy

BinaryClassificationEvaluator uses scores and known labels.

from pyspark.ml.evaluation import BinaryClassificationEvaluator
evaluator = BinaryClassificationEvaluator(
    labelCol="label", rawPredictionCol="rawPrediction",
    metricName="areaUnderROC"
)
roc_auc = evaluator.evaluate(predictions)
print("Validation ROC AUC:", roc_auc)
Positive scorerawPrediction by defaultROC curveTrue vs false positiveratesAreaRanking discriminationsweepscorethresholdsintegratecurve
INF2006 · BIG DATA 035

Choose metrics for the decision costs

Rare positives can make accuracy misleading.

5 min · A model predicts no churn in a 95:5 dataset. What does 95% accuracy hide?
PrecisionHow many alerts are correct?RecallHow many positives does themodel detect?PR AUCRanking under imbalance
INF2006 · BIG DATA 036

Precision asks whether alerts are correct

Precision measures correctness among predicted positives.

INPUTModel alerts4 positive alerts3 truly positive1 false positiveOUTPUTPrecision3 / (3 + 1)Precision = 0.75Count alerted positivesTP = 3FP = 1
INF2006 · BIG DATA 037

Recall measures the proportion of detected positives

Recall measures coverage among actual positives.

INPUTActual positives5 true positives exist3 detected2 missedOUTPUTRecall3 / (3 + 2)Recall = 0.60Count actual positivesTP = 3FN = 2
INF2006 · BIG DATA 038

A PR curve traces precision and recall together

PR AUC summarises a curve. It does not measure accuracy at one threshold.

INPUTRanked scores0.9: positive0.8: negative0.6: positiveOUTPUTOperating pointsPrecision changesRecall changesArea summarisesrankingSweep score thresholdsHigh threshold: fewalertsLower: more coverage
INF2006 · BIG DATA 039

Model selection

How do we select model settings without using the final test set?

INF2006 · BIG DATA 040

Validation chooses hyperparameters

Do not use test data to select settings.

from pyspark.ml.tuning import ParamGridBuilder
grid = ParamGridBuilder() \
    .addGrid(lr.regParam, [0.0, 0.1]) \
    .addGrid(lr.elasticNetParam, [0.0, 1.0]) \
    .build()
# Four candidate settings; evaluate on validation only.
Candidate settingsregParam, elasticNetParamValidation evidenceCompare same metricSelected modelFinal holdout oncefit andcomparecandidatesselectsetting
INF2006 · BIG DATA 041

Compare against a simple baseline

A more complex model must justify its additional cost.

BaselineSimple rule or classifierAlternativeRandom forest or added featuresDecisionValidity, metric, runtime
INF2006 · BIG DATA 042

A baseline provides a reference for model comparison

A trivial rule can reveal misleading accuracy.

INPUTValidation casesSame featureavailabilitySame temporal boundaryOUTPUTBaseline evidenceDeclared metricFailures examinedSimple referenceMajority / prior ruleOr supplied simple model
INF2006 · BIG DATA 043

Change the model within the same evaluation boundary

Additional complexity must justify its extra cost.

INPUTTraining dataSame rowsSame allowed featuresOUTPUTComparable resultSame held-out casesSame metric + costAlternative modelChange one model choiceTune on validation only
INF2006 · BIG DATA 044

Choose with error costs and evidence

Selection requires a reason, not only a higher headline score.

INPUTCandidate evidenceMetric valuesError casesRuntime + complexityOUTPUTSelected approachValidationjustificationFreeze choicesFinal test onceReview the decisionWhat mistakes matter?Is evidence reproducible?
INF2006 · BIG DATA 045

Machine learning lab

What evidence makes the reported model result reproducible and valid?

INF2006 · BIG DATA 046

Report the metric the lab actually computes

Check the metric in the unchanged lab code.

PrepareTypes, labels, vectorsTrainTraining-only learned stagesReportROC AUC, split, limitations

Valid evaluation is part of the pipeline

The result is useful only if the evaluation boundary is valid.

01DefineUnit, timestamp, positive class02FitTraining-only learned stages03EvaluateDeclared metric, final holdout
INF2006 · BIG DATA 048

Hive and
distributed SQL

Catalog / SQL / Partitions

A table is a view over distributed data.

INF2006 · CLOUD COMPUTING AND BIG DATA001

Hive gives shared files a SQL interface

A table definition connects names and types to stored data.

FilesHDFS or S3Table metadataSchema and locationSQL questionAggregate and filterresolve tablelocation reference
INF2006 · BIG DATA 002

Hive architecture

How does the catalog connect SQL names to files and execution?

INF2006 · BIG DATA 003

The catalog describes books and their locations

Keep metadata, execution and storage distinct.

AnalystAsks a questionDriverPlans the requestMetastoreCatalogdescriptionsStorageActual recordsSQLrequestmetadatalookupdescribeslocation
INF2006 · BIG DATA 004

The execution engine reads the records

The driver plans the query. Workers execute the distributed query.

ClientSQL statementDriverParse, compile, optimizeMetastoreSchema and partitionsEngineTasks on workersFilesHDFS / S3 bytesSQLmetadatarequestexecution planrecord reads
INF2006 · BIG DATA 005

EMR Hive normally uses Tez

Check the release and session settings. Do not assume that Hive uses Spark.

-- Read the actual Hive session setting.
SET hive.execution.engine;
-- Record EMR release and Hive version too.
HiveSQL compiler and servicesTezEMR default engineSpark SQLSeparate Spark query path
INF2006 · BIG DATA 006

A metastore entry defines how to read stored data

Schema, format and location must agree with the files.

5 min · The table location is correct but movie_id is NULL in many rows. Which contract fields would you inspect?
Columnsuser_id INT · rating INTFormatTab-delimited textLocation/user/hive/data/movielens
INF2006 · BIG DATA 007

Column order and types interpret each field

A correct location cannot repair a wrong declared schema.

INPUTTSV bytes196 TAB 242 TAB 3Fourth field:timestampOUTPUTParsed values196 | 242 | 3 | tsBad cast may becomeNULLDeclare four fieldsuser_id, movie_idrating, rating_ts
INF2006 · BIG DATA 008

The delimiter must match the bytes

Format metadata tells the reader how to decode records.

INPUTRaw line196<TAB>242<TAB>3<TAB>tsNo header rowOUTPUTFour fieldsuser / movie / rating/ tsWrong separator: badparseRead tab-delimited textSplit on TAB, not commaApply declared types
INF2006 · BIG DATA 009

A table location points to a dataset directory

Keep unrelated or incompatible files out of a table directory.

INPUTRegistered location/user/hive/data/movielens/ratings.tsv insideOUTPUTQueryable rowsMetadata is a pointerSource files stayseparateResolve table filesRead eligible filesApply one table schema
INF2006 · BIG DATA 010

Schema on read

How does a table schema interpret data that is already stored?

INF2006 · BIG DATA 011

Schema-on-read interprets existing data

Declared types do not correct malformed records.

Raw line196 TAB242 TAB3 TAB881250949Declared fieldsINT, INT, INT, BIGINTRead resultTyped columns or parseissuesparseviadeclaredformatexposetypedcolumns
INF2006 · BIG DATA 012

Table lifecycle

What happens to the stored data when we remove a table?

INF2006 · BIG DATA 013

Managed tables include a data lifecycle

Dropping a managed table normally removes its data and metadata.

MetadataTable definitionManaged storageHive lifecycleDROPNormally removes bothlifecycleincludesfilesDROPnormallyremovesboth
INF2006 · BIG DATA 014

External tables separate registration from ownership

Default DROP removes metadata. Separate controls must protect files from deletion.

MetadataHive registrationExternal filesIndependent lifecycleDefault DROPDROP removes metadata andretains fileslocationpointerremoves registration
INF2006 · BIG DATA 015

Check deletion behavior before changing data

Table type alone is not the complete answer.

5 min · Does CREATE EXTERNAL TABLE guarantee source files can never be deleted?
Managed scratchHive-owned outputExternal sourceShared ratings filesPurge enabled?Check properties and version
INF2006 · BIG DATA 016

Stage the lab data

Where must the source files be before Hive can read them?

INF2006 · BIG DATA 017

Stage files before declaring the table

Local download and HDFS storage are separate locations.

hadoop fs -mkdir -p /user/hive/data/movielens
hadoop fs -put ratings.tsv /user/hive/data/movielens/
hadoop fs -ls /user/hive/data/movielens/
Local ratings.tsvDownloaded on primary nodeHDFS directoryUploaded data bytesHive tableRegisters HDFS locationuploadbytesregisterlocationreference

Table definitions

How do we describe the file format, columns and location in SQL?

INF2006 · BIG DATA 019

Declare the TSV schema and location

One directory can contain several compatible part files.

CREATE EXTERNAL TABLE ratings (
  user_id INT, movie_id INT,
  rating INT, rating_ts BIGINT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION '/user/hive/data/movielens/';
Four fieldsUser, movie, rating, timestampTab delimiterNo header rowLocationExisting HDFS directory
INF2006 · BIG DATA 020

Validate a table before aggregating

Inspect metadata, sampled values and row counts.

DESCRIBE FORMATTED ratings;
SELECT * FROM ratings LIMIT 5;
SELECT COUNT(*) FROM ratings;
SELECT MIN(rating), MAX(rating) FROM ratings;
DESCRIBEDefinition and locationSampleExpected column valuesCountExpected scale
INF2006 · BIG DATA 021

HiveQL

How do SQL queries turn the stored records into useful summaries?

INF2006 · BIG DATA 022

Aggregation answers the source business question

Top rated by count is different from highest average rating.

SELECT movie_id, COUNT(*) AS rating_count
FROM ratings
GROUP BY movie_id
ORDER BY rating_count DESC, movie_id
LIMIT 10;
RowsOne rating event eachGROUP BYOne group per movieORDER + LIMITMost-rated tengroup rating eventssort grouped counts
INF2006 · BIG DATA 023

WHERE filters rows. HAVING filters groups.

Apply thresholds after calculating aggregate values.

SELECT movie_id, COUNT(*) AS n, AVG(rating) AS avg_r
FROM ratings
WHERE rating BETWEEN 1 AND 5
GROUP BY movie_id
HAVING COUNT(*) >= 100 AND AVG(rating) > 4.0
ORDER BY avg_r DESC, movie_id
LIMIT 5;
WHEREKeep valid input ratingsGROUP BYCount and average per movieHAVINGKeep qualifying groupsaggregateretainedrowsfilteraggregateresults
INF2006 · BIG DATA 024

Choose the right filter location

Explain the intermediate relation, not only the syntax.

6 min · Find movies with at least 100 ratings and a mean above 4. Where do those conditions belong?
Raw ratingsIndividual eventsGrouped moviesCounts and meansSelected moviesThresholded output
INF2006 · BIG DATA 025

Joins connect identifiers to related data

Key uniqueness determines whether rows multiply.

SELECT m.title, COUNT(*) AS n
FROM ratings r
JOIN movies m ON r.movie_id = m.movie_id
GROUP BY m.title
ORDER BY n DESC
LIMIT 10;
Ratingsmovie_id repeatedper eventMoviesOne title permovie_idJoined eventsRatings with titlesevent rowsmatching movie_id
INF2006 · BIG DATA 026

A left join can retain unmatched events

A WHERE filter on the right side may remove them again.

Left ratingsPreserve these rowsMissing movieRight columns become NULLWHERE right fieldMay discard NULL rows
INF2006 · BIG DATA 027

A left join starts by preserving left-side events

Every left row remains, including rows without a right match.

INPUTRatingsmovie 1: rating5movie 9: rating3OUTPUTJoined rows1 | 5 | title A9 | 3 | NULLLEFT JOIN moviesMovie1 has title AMovie9 has no match
INF2006 · BIG DATA 028

An unmatched lookup produces NULL right fields

NULL can describe a missing match rather than a missing left record.

INPUTJoined evidencemovie1 → title Amovie9 → NULLOUTPUTQuality signalUnmatched countrecordedCoverage issueinvestigatedInspect the missingmatchNo catalog record for9Preserve event identity
INF2006 · BIG DATA 029

A right-side WHERE filter can discard unmatched rows

SQL filtering can undo the preservation you expected from a left join.

INPUTLeft-joined rows1 | title A | year20209 | NULL | NULLOUTPUTFiltered resultMovie1 retainedMovie9 discardedWHERE year = 2020Matching row passesNULL condition is unknown
INF2006 · BIG DATA 030

Complex data

How can Hive expose fields inside nested JSON records?

INF2006 · BIG DATA 031

STRUCT gives nested JSON named fields

The SerDe must match the file format and installed release.

CREATE EXTERNAL TABLE posts (
  id BIGINT, text STRING,
  author STRUCT<name:STRING, followers:INT>
)
ROW FORMAT SERDE
  'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION '/data/teaching/posts/';
JSON recordNested author objectSerDeInterpret fieldsSTRUCTauthor.name · followers
INF2006 · BIG DATA 032

Dot notation selects a nested field

The result can remain a simple SQL relation.

SELECT text, author.name
FROM posts
WHERE author.followers > 1000000;
author struct{name:Ada, followers:1200000}Filterfollowers >1000000OutputAda + post textevaluate nestedpredicateemit selected fields
INF2006 · BIG DATA 033

Storage layout

How can directory partitions reduce the files a query must consider?

INF2006 · BIG DATA 034

Partitioning makes directory exclusion possible

Partition keys describe the directory layout. They do not repeat in the row schema.

CREATE EXTERNAL TABLE sales (
  order_id STRING, amount DOUBLE
)
PARTITIONED BY (dt STRING)
STORED AS PARQUET
LOCATION '/data/teaching/sales/';
Row schemaorder_id, amountPartition keydt STRINGDirectoriesdt=2026-10-01 / …stored rowsdirectory key
INF2006 · BIG DATA 035

Folders need corresponding catalog partitions

Creating a partitioned table does not register every existing directory.

ALTER TABLE sales ADD IF NOT EXISTS
PARTITION (dt='2026-10-01')
LOCATION '/data/teaching/sales/dt=2026-10-01/';
SHOW PARTITIONS sales;
Existing folderdt=2026-10-01ADD PARTITIONRegister key and locationMetastorePartition available toplannerlocationreferencemetadataupdate
INF2006 · BIG DATA 036

Query planning

How do partition filters let the engine skip unrelated directories?

INF2006 · BIG DATA 037

Pruning skips unrelated partition locations

Check the plan and measured bytes before you claim faster execution.

EXPLAIN
SELECT SUM(amount) FROM sales
WHERE dt = '2026-10-01';
QueryWHERE dt=2026-10-01October01Eligible directorySeptember30Excluded directoryreadeligiblepartitionprune: do not read
INF2006 · BIG DATA 038

Match partition keys to the query pattern

A catalog and storage layout must work together.

5 min · Would partitioning by unique order_id improve monthly SUM(amount) reports?
By dateUseful for daily reportsBy order_idMany tiny directoriesNo date predicateDate partitions may all remain
INF2006 · BIG DATA 039

Integration

How can several analytical tools use the same data and metadata?

INF2006 · BIG DATA 040

Shared metadata enables several analytical tools

Sharing a catalog does not make every engine execute identically.

CatalogAgreed schema andlocationsHive / TezSQL warehousequeriesSparkDataFrames and MLBI clientReports through aquery servicetablemetadatatable metadataSQL via service
INF2006 · BIG DATA 041

Hive lab preparation

Can we trace each lab query from SQL through metadata to the source files?

INF2006 · BIG DATA 042

Follow the query from SQL to files

Keep screenshots and your reflection. Terminate the cluster.

01Stage + declareRatings TSV and external DDL02Query + inspectCounts, averages and groups03Explain + reflectMetadata, execution, storage