Big Data Engineering hat sich in jüngster Zeit zu einem eigenständigen Beruf entwickelt. Und selbst große Unternehmen verwechseln sehr oft, was dieser Spezialist tut, was seine Kompetenzen sind und warum er überhaupt in der Organisation ist.
, Big Data Engineer, Data Analyst Data Scientist. , . , , .
Big data engineer
, , . data science.
Data engineer — , , . , .
: e-commerce, , , — , .
, «» . IoT- . , . , . , , .
, :
. BDE . . ETL (extract, transform, load) — , . . Kafka, Apache Spark, Storm, Flink, Google Cloud Azure.
. — -. — , .
. , , . , .
. - , , .
: , .
Data Engineer
;
SQL NoSQL . : MySQL, PostgreSQL, MongoDB, Oracle, HP Vertica, Amazon Redshift;
ETL- (BM WebSphere DataStage; Informatica PowerCenter; Oracle Data Integrator; SAP Data Services; SAS Data Integration Server);
Amazon Web Services, Google Cloud Platform, Microsoft Azure;
Apache SQL- ;
(Python, Scala, Java).
-, , , .
Data Engineer , -. , , . - , — , .
Business Intelligence, .
, Data Engineer Data Scientist , . — , . — .
:
. . BDE , , . hh.ru, , 768 .
Big Data Engineer , Data Scientist. , . — ( ). , .
— , . . .
. — . . -, , .
, .
Spark — , , .
Flink — Spark / , .
Kafka — , . ElasticSearch — , Apache Lucene.
PostgreSQL — .
Redshift — / AWS.
BD Engineering . , 1–2 . Python Scala — .
. , . — Data Scientist Data Analyst, - , . . .
Data Engineer
- . Python Scala, Big Data. , , 50/50. Java — .
SQL . Data Engineer , : Data Analyst, Business Analyst, Data Scientist. - 1–2 .
, .
Big Data Engineers . Data Science Data Analytics, , Devops-. Python Scala, .
. Dice Tech Job Report 2020, Data Engineering — 2019 50 %. : 3–5 %.
2020 , . 24,8 %. .
, Data Engineering Data Engineering Data Science . , , .
