Just about one year ago, I was trying to make the distinction between data science and data engineering. But in the past year a new job role has become very popular (at least I noticed it in job postings and media). The machine learning engineer. But what is the difference between this new job role and data engineering? Or is it just a different name for the same thing?
To recap the last post in short statements:
Data science is the discipline of extracting insights from data.
Data engineering is the discipline of making data accessible.
Fit the curve
Data engineers are often quite close to infrastructure topics and data pipelines. The most data engineers are working with big data technologies, like hadoop, elasticsearch and noSQL etc. In general data engineers do not necessarily deal with machine learning, but build pipelines, that enable machine learning for distributed data/computing, make model deployment scalable and robust. Automation is also a big part of data engineering.
The machine learning engineer obviously works with machine learning. Like a data scientist he has to know ML algorithms and techniques, but from the software engineer’s perspective. It does not really matter if the data comes from HDFS or a file stored on disk, ML engineers must deal with all kind of data sources.
Machine learning engineers build software related to ML problems, which includes the process of training, validation and deployment of models into production systems. Furthermore, cloud computing (AWS, GPU acceleration), non-academic languages (C++/Java) and specific frameworks (sci-kit learn, tensorflow, spark,..) are often part of the job. Its possible that algorithms or implementations are not suitable for a specific problem and have to be optimized or customized. Because of the engineering nature it´s likely, that the job is more about applied machine learning, rather than research.
If data engineers are closest to infrastructure, data scientists are closest to statistical modeling, then ml engineers are closest to software engineering.
Machine learning engineering is the discipline creating machine learning software products
Now we have a definition. But there is still some, let´s call it variance, out there.
- AI Engineer & Analyst
- Data engineer and machine learning specialist
- AI and data engineer
- Data Scientist / Machine Learning Expert
- Software Development Engineer – Deep Learning AWS AI
- Machine Learning Developer
- Data Scientist / Machine Learning
- Data / Machine Learning Engineer
- Data Engineer & AI Expert
- Applied Machine Learning Engineer
- Software Engineer: Machine Intelligence