Why Data Engineering?
AnalystToEngineer is based on my journey of transitioning from Data Analyst to Data Engineer. Here is why I am making that transition
When I talk to people about transitioning into Data Engineering - especially those outside the field - reactions are often mixed. Some are optimistic, drawn by the potential for higher salaries, while others are more apprehensive due to the rise of AI and its potential to disrupt the field.
Both perspectives hold some truth. Data Engineers generally earn more than Data Analysts. AI’s will disrupt industries and jobs, including Data Engineering.
But for me and why I’m looking to break into Data Engineering, let me share my thoughts on this.
Data Engineering is Growing
In the last few months, I’ve noticed some exciting indicators:
1️⃣ Python’s Rising Popularity
For the first time, Python was named the most popular coding language on Github. While its dominance is largely driven by AI-related projects, Python also plays a critical role in Data Engineering. It also allows for popular frameworks like Airflow and Spark (PySpark) making Python essential for data professionals.
2️⃣ Job Growth in Data Engineering
According to Avery Smith’s “State of Data Jobs in 2025” article, Data Engineering job listings have shown the most growth among data-related roles since ChatGPT launched in late 2022. This is a promising trend for anyone considering entering the field. Whats interesting is the clear growth in Data Engineering compared to all other titles in the data family.
To stay on top of this, I’m working on a personal project to build a pipeline that tracks job listings, skills, and salaries for data roles in Australia which I look forward to sharing more on in upcoming posts.
Discovering a passion for ‘Data Engineering-like’ work
Although I’ve never been a full-time Data Engineer, many of the responsibilities I’ve taken on as a Data Analyst align closely with the role. One of the best aspects of being a Data Analyst is the exposure to diverse tasks - and I’ve found that the ‘Data Engineering-like’ ones are the most fulfilling.
Take my time at AGL Energy, for example:
I was part of a team that managed a data mart, consisting of views, tables, and stored procedures in a data warehouse.
Orchestration was handled with SSIS, Task Scheduler and Python for sequencing jobs to transform raw source data into tables ready for business use.
Whether it was maintaining pipelines, resolving failures or implementing changes, the focus on quality and requirement for structure & documentation matched my working style.
It wasnt always smooth. Jobs would break, changes were needed often and quickly but overall it was something I enjoyed upon reflection.
Making Magic Happen as a Data Engineer
During my time at Uber, I worked on moving data from source to business value. This transformation felt like making magic happen especially when you see the data that you are making available being applied.
I ingested data from multiple formats (e.g., SFTP and APIs) and at varying intervals.
I transformed this into something useable and loaded it into a data warehouse for dashboards, reporting, and operational use.
One unique challenge was handling unstructured image data. This involved the use of cloud blob storage to host images and storing accessible URLs in the database.
Seeing that data come to life from an external source into the Uber app was an incredibly rewarding feeling.
Of course, there are aspects of Data Engineering I wont enjoy, such as being on-call. But for now, I’m energized with the direction my career is heading.
What about you? Have you considered a move into Data Engineering, or are you already on this path? I’d love to hear your thoughts.