Data Science Wire

Why does the same Spark job suddenly get much slower when the data shape changes even if total volume barely moved?

Reddit r/apachespark6d4 min read

we've got a job that runs fine for weeks and then randomly triples in runtime even though row counts barely changed. eventually traced it to cardinality and skew shifting under the hood, but it's hard to plan capacity or set SLAs around something that isn't tied to volume at all. the annoying part is our monitoring was built around volume as the main signal, so nothing flagged this until the job had already blown its window. makes capacity planning feel like guesswork when the thing driving runtime isn't the number we're tracking. is this just a spark reality once you're at a certain scale, or

Read the full story at Reddit r/apachespark

More in Data Engineering