How OpenAI's Security Team Turned a Streaming Pipeline Into Its Own Latency Watchdog with SDP ForEachBatch
Reddit r/databricks1w4 min read
https://community.databricks.com/t5/technical-blog/one-pipeline-any-destination-the-foreachbatch-sink-in-spark/ba-p/163780 The blog highlights OpenAI's usage of forEachBatch inside SDP to power their security platform pipelines. submitted by /u/Much-Neat-6273 [link] [comments]
