Data DevOps Engineer

For eCS new global BI/Data analytics solution based on Microsoft Azure Synapse/DataBricks, we urgently look for an established Senior Data Engineer with following qualifications:

At least 3 years of track record proven hands-on working knowledge on (in prioritized order)

  1. large-scale Data processing technologies such as Apache Spark utilizing Scala and PySpark (must have), Hadoop Stack or similar
  2. stream processing technologies such as Spark Structured Streaming (must have), Kafka (must have), Apache Flink, Apache Storm or similar
  3. developing DWH solutions on NoSQL and relational databases

Candidates need to prepared to partake in an expert interview in which hands-on experience will be assessed on architectural level (understanding of the technologies mentioned above) as well as on engineering level. The candidate will be prompted with problem solving tasks and asked to verbally reflect on how to complete the tasks at hand utilizing the technologies depicted above.

Proven project experience in hands-on engineering on Azure Data stack: Spark/Databricks, ADLS Gen2/Blob, EventHub/Kafka, ADE, SQL, Azure Synapse/SQL DWH, Python/Scala programming, Azure functions. Responsibilities include (most important tasks listed):

  • Develop central ETL routines + stored procedures, mostly in Apache Spark
  • Develop Stream processing routines in Spark Structured Streaming and/or Apache Flink consuming Data from Kafka, Azure Eventhubs
  • Develop DWH solutions on Azure Cosmos DB, Azure Synapse Analytics, Azure SQL and similiar
  • Admin + Maintain (run/ops) of central data platform on Azure Cloud + ensure data consistency
  • Use case specific data pipeline design and development if needed

Azure DevOps/terraform experience (nice to have); responsibilities, e.g.:

  • Maintain/Configure Azure DevOps CI/CD Pipelines, Repositories & Artifactories
  • Develop Build/Deploy (Pipeline) + Terraform (IaC) Scripts & Tools (Automation) based on CorpDL templates
  • Design & Implement Unit & Integration Testing for pipelines & IaC deployments

Practicall experience with public cloud and Big Data technologies (primarily, Azure based). Strong DevOps attitude and skills (CICD, automation etc)

Long term assignment at least till the mid of 2022