Data Science Wire

help in building AWS pipeline (100k records to be processed by an api)

Reddit r/ETL1d4 min read

I have an API (built by another team) that I need to send \~100,000+ records through, and I'm trying to figure out the right pipeline architecture. Looking for a sanity check / better ideas from people who've solved similar problems. The API: • You POST a JSON payload and it immediately responds with status: initiated (i.e. it's async — the response does NOT mean the job is done) • It processes the payload in the background and takes \~15-20 seconds per record • Once done, it writes the result directly to a Postgres table (no webhook/callback, and no clear "check status" endpoint). We can also

Read the full story at Reddit r/ETL

More in Data Engineering