How I’m Breaking into Data Engineering
And how any pivot can be made using the the same steps
Career changes can be daunting and feel like you are starting from zero.
It can even feel like going backwards because previous work experience may seem irrelevant.
When I set my sights on Data Engineering, I had no prior experience as one.
What I did have though was a clear intention. I wanted to move into Data Engineering having spent most of my career as a Data Analyst. From consuming data to building the pipelines and infrastructure that enables it.
In this article I share the approach taken to prepare for the switch across 3 pillars.
A repeatable approach for anyone in a similar situation, whether its a mid-career transition, just starting out or someone with several years of experience.
It doesnt include the job application process, that content is for another time.
Pillar #1 Learning
Know the Foundations
A foundation of knowledge is needed before breaking into any new field.
Thanks to the internet there are so many ways this can be achieved - Higher Education, Certificates, Youtube, Podcasts, Blogs, Books, AI & more.
The foundations I have are:
Completed the IBM Data Engineering Professional Certificate in Coursera.
Completed a Data Engineering Bootcamp on DataExpert.io.
Read the Fundamentals of Data Engineering O’Reily book.
Do these overlap? Yes. Then why invest the time, effort and money in each? Because they cover Data Engineering at different depths and formats.
The IBM certification is a solid beginner friendly, self-paced option with plenty of structure and guidance.
If practical hands-on work that requires curiosity & problem is what I am after then the Dataexpert.io bootcamp is more relevant.
What about conceptual depth agnostic of any tools? The O’Reily book is best.
Variety of learning is important and indexing too much on a single type can be stale. Collecting every single certificate like badges is neither an effective use of time or a stimulating way to learn.
Pillar #2 Applying
Build Personal Projects
Learning provides the knowledge but applying it is where the magic happens. The job title is not needed when it comes to building personal projects.
Projects I have created include:
Solving a real world problem or question → Like understanding more about Data Engineering jobs, salaries and skills so I built a job listing pipeline.
Getting hands on experience with modern tools on topics of interest → such as using Google Bigquery to understand more about the most commonly used Python libraries
Exploring alternative technology → By building something familiar like a Pokedex hosted on Streamlit community cloud.
Creating a portfolio shows an ability to apply the knowledge from pillar 1.
This doesnt mean mass producing 10 different projects. 2-3 high-quality builds that can be elaborated on is sufficient enough to satisfy this pillar.
Pillar #3 Sharing
Being visible on the journey
This is all about documenting and showing both the intent & actions publicly.
Learn something new and interesting? Those can be shared with a professional network.
Build a project with interesting results? Write an article diving deeper into the methodology & insights.
This isn’t just limited to pillars 1 & 2, its also about about being active in building a personal brand around the transition.
Examples for me includes sharing about:
All aspects of the journey such as summary notes, learnings & insights from studies or a portfolio project on Linkedin and Substack
My involvement and take aways from being a part of data communities, events & conferences.
How does this help my transition? Its a clear signal to my network, shows how passionate I am, keeps me accountable all while keeping up to date with the latest developments in the industry.
Yes there are lessons learnt along the way with branding - not many employers will hire “Aspiring” Data Engineers since they are still learning.
However there is a net gain when to comes to sharing because it opens up more opportunities then it closes.
Final Thoughts
Journeying into Data Engineering is still unfolding for me and its based on the 3 pillars of Learning, Applying and Sharing. These pillars have given me a starting point and if you are also planning a pivot or career change, whether its in Data Engineering or any other field then I hope this also helps with giving clarity on what those first few steps look like.