When training AI model how do you decide whether to invest more in faster GPUs or better data pipelines?

Its easy to think that faster GPUs will always speed up the training process but if parts like data loading, preprocessing, storage speed or communication between systems become the slowest parts then adding more powerful hardware might not give the expected results. The balance between different options becomes more complex when comparing services like AWS, Google Cloud, CoreWeave and Yotta Labs. The option that offers the lowest cost per GPU hour isn't always the most cost effective way to complete the training. Do you guys start by testing the whole training process to find the bottlenecks