Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs
Reddit r/dataengineering1w4 min read
Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually. submitted by /u/rgancarz [link] [comments]
