What's Silently Breaking Your Automation?