You passed the fundamentals round. You know what a DataFrame is, you understand lazy evaluation, and you can explain the difference between repartition and coalesce. You got the mid-level role.
Then the senior interview comes — and you hit a wall.
The questions shift from “what is X?” to “your Spark job is taking 6 hours and the SLA is 2 hours. Walk me through how you’d fix it.”
That’s what this article is for. These are the concepts that separate engineers who use Spark from engineers who optimizeSpark. Let’s get into it.
1️⃣ Broadcast Join vs. Shuffle Hash Join vs. Sort-Merge Join
The Trap Answer
“Use broadcast join for small tables.”
Every candidate says this. What interviewers want to know is: how do you decide, and what happens if you get it wrong?
