Remote workers have all the time in the world — but fewer boundaries than ever. — I Would Give Anything To Do Nothing (For 5 Minutes) It’s 6-something on a Wednesday evening and I’m working 500+ miles from my usual desk. I clock out after a particularly tough day of data engineering and play on my phone for about 5 minutes before my “office” (my wife’s childhood bedroom) door bursts open and one of my many…

A precautionary tale for data scientists using paid API subscription services in personal projects. — How I Ended Up With A $300 Bill For An API Looking at my RapidAPI bill recently made me feel like an exasperated parent. Specifically, I felt like the parent of a child who didn’t grasp the concept of micro-transactions, those seemingly trivial in-app purchases that always add up. Except, as a professional data engineer, I felt a bit worse. I…

Why your first instinct to convert raw data into a data frame is probably wrong. — To receive this story and other content first, consider joining my email list. If you’re like me your very first interaction with data was likely through one command. read_csv. This simple function, universal for data scientists working in every language from Python to R, is fundamental to processing and manipulating…

How I used python to help a Wall Street banker pick stocks (part II). — This is the second part in a multi-part series. See part I below: How I’m Using Data Engineering To Help A Wall Street Banker Pick Better Stocks (Part I) How a conversation between friends turned into a potentially lucrative data side project (with code walkthrough).medium.com Data Science Is Cool, But It’s Not Magic Being immersed in the data science world (at work and in online communities), sometimes you forget how impressive and, frankly, cool, a well-conceived data product can be.

How a conversation between friends turned into a potentially lucrative data side project (with code walkthrough). — “Random Question — Can You Get Earnings Data On Every Publicly Traded U.S. Company?” “Hey man, is there a way to automate this?” I look up from this weekend’s bed (AKA my friend’s couch) to see my childhood friend, an investment banking analyst, copying and pasting a company’s earnings data into an Excel document on his dual-screen desktop.

Don’t get too invested in the conception of your project that you forget this resume-boosting component. — My Work Confused A LinkedIn Connection — And Jump-started An Important Conversation Recently, a LinkedIn connection asked a question I had not expected or received. In so many words they asked why some of the projects featured in my story about creating and compiling a data science portfolio included cloud-based pipelines.

How to create, evaluate and apply test data to data pipeline builds. — It wasn’t until about a year into my position as a data engineer that I realized that test data was not a requirement. It was a luxury. Initially, I assumed nearly all projects would be accompanied with some kind of data that I could use to “sanity test” my builds…

Understand when to stream data, how to configure a pipeline and learn from my mistakes building streaming pipelines. — The Forgotten Streaming Library Hidden In Python When you think of building streaming data pipelines you likely think of using a library like Apache Kafka that is tailor-made for streaming. Cloud platforms like Google Cloud Platforms (GCP) and Amazon Web Services (AWS) support the streaming of data into their respective SQL environments. GCP, in particular, even provides…

Isolation + self-doubt = panic. — Imposter Syndrome Leads to Remote Work Anxiety In 2021, within 30 days of each other, 3 things happened: My now-wife and I were in the final stages of planning our wedding, I started a new full-time job and an immediate family member nearly died. Each event brought mental stress, compounding a battle I had quietly been fighting…