Why developing business knowledge is as important as developing technical skills. — Amassing the technical skills and confidence to apply for and thrive in data engineering roles can feel daunting, but such a heavy focus on up skilling neglects the importance of developing domain knowledge. Be the Master of Your Domain… Knowledge If you are an aspiring data professional reading this, good news: We’re in the middle of a…

How a data engineering job can help you identify strengths, interests and skill gaps. — Don’t Undervalue Your Data Skills Perhaps the biggest mistake I made as a data science student coming out of graduate school was to limit myself to one kind of job application. As I approached my final semester I realized that while my programming skills were solid, communication skills were honed from journalism and my domain…

How to build queries that analyze Cloud Logging data. — A Meta Query Language LQL can be used in the Logs Explorer to fetch real-time data on Google Cloud products like Cloud Functions and Virtual Machines as well as non-GCP resources like resources connected to Amazon Web Services’ accounts. If you’ve never accessed the Logs Explorer within your GCP project, you can navigate to…

Steps you must take to prepare to learn and apply SQL. — Earlier this year a friend came to me, admitted that they were unhappy with their current career trajectory, and that they were looking for a change. They were considering learning some ‘hard’ or technical skills to become more competitive in the job market. Did I have any advice? My advice…

Although unassuming tasks, revising code and periodically rethinking infrastructure can make a big impact. — The Data Maintenance Worker Data engineering, as a profession, flies under the organizational radar. Although our data peers know us as the janitors or plumbers of data science, it can sometimes be difficult to see, immediately, how data engineering work impacts an organization or your data team at large.

The subtle ways data engineering can become over engineering. — Over-Engineering in Software Engineering Tinkering, tweaking and adjusting are all innocent verbs that describe a troubling tendency of both experienced and inexperienced software engineers to improve upon that which doesn’t need improvement. Typically, over-engineering is a phenomenon that tends to impact younger programmers (myself included) who may be only a few years removed from…

Forget passive income; focus on deepening knowledge, building credibility and growing awareness for the sometimes-misunderstood data field. — Having been a tutor, I’ve experienced, firsthand, the benefits of developing knowledge through instruction and explanation. Even though I never became a teacher (though I’ve worked with plenty of amazing instructors), I’ve seen how much of an impact sharing feedback or wisdom can have on an individual’s personal and professional…

Very few data engineering degree programs exist, perpetuating a vicious cycle of un-credentialed and “unqualified” candidates. — A Chronic Data Engineering Shortage As the data industry booms, we are in the middle of a widely publicized data engineer shortage. By some estimates, we are more than 5 years into this striking disparity in talent supply and employer demand. The reason, as I’d like to argue, isn’t merely the fact that data science…

How data engineers can identify, reconfigure and address malformed CSV files in batch pipelines. — The following is dedicated to my past self, who tried, through brute force and willpower, to fix malformed files. I hope you glean something useful so that you avoid a similar situation and save yourself some wasted working hours. The Dreaded Malformed or Corrupted File Error While all Python errors are annoying on some level, lately the…