Renting a moving truck to grab a pizza? Feedback on my OSINT pipeline architecture
Reddit r/ETL6d4 min read
Working on a personal pipeline that pulls public signals from a few sources (earthquakes, internet infrastructure status, disaster alerts, markets, etc.), compares each one against its own historical baseline, and runs on AWS with cache invalidation on every update. The goal from the start was for it to run on its own at the lowest possible cost, without buying hardware I'd have to babysit like a pet. (I already have a cat for that.) No boxes to maintain, no drives to worry about. Question for people who've done this longer than me: does this sound reasonable, or is it over-dimensioned? It fee


