Overcoming The Final Hurdle of Data Automation With Fewer Failures

Learn the components of data pipeline production to take your ETL build from code to cloud with automated, actionable results.

Don’t have anything to deploy yet? Create a job-worthy data portfolio. Learn how with my free project guide.

The Development Practice You Take For Granted

I’m the embodiment of the meme in which a developer spends hours automating a relatively simple task. In other words, while much of the world is increasingly apprehensive of replacing processes with AI, I’m still pro-automation.

Image courtesy of starecat.com.

And while I’ve developed some pipelines outside of work to serve my own needs or to help out a friend, I still struggled with one very important aspect of each ETL build.

If you’re reading this, I imagine you might struggle with the same issue.

Deployment.

To be clear, at work in nearly two years I’ve written thousands of lines of code, created probably 50-ish pipelines and written CI/CD processes ranging from cloud function deployment to Docker image updates.

When I wanted to replicate some of these processes with my own builds, I…