Senior Data Engineer

Salary:

Negotiable

Opportunity type:

  1. Interim

Location:

Newcastle upon Tyne, Tyne and Wear

Specialism:

  1. Disruptive Technology

Date posted:

22 Aug 2019

Opportunity reference:

SH1506_1566470286

Opportunity description

Key Responsibilities:

  • Lead the design, implementation, and continuous delivery of pipelines using distributed Azure based big data technologies supporting data processing initiatives across batch and streaming datasets
  • Responsible for development using Scala, Python languages and Big Data Frameworks such as Spark, EMR, Kafka, Storm, Jenkins, Jfrog Artifactory and DataBricks
  • Provide oversight and direction on deployed Azure platform components
  • Process and manage high volume real time customer interaction streams
  • Support the data analytic team in the creation of data models and product/propositions.
  • Development and implementation of data flows and ETL's to ingest data, using both streaming and batch technologies from multiple sources
  • Design and development of the deployment processes using continuous integration
  • Provide technical review of the solution and identify project risks and estimate development time for such solution.
  • Support entire development process and conduct code reviews.

Requirements:

  • Bachelor's Degree or higher in Computer Sciences or similar
  • Development experience with Azure services
  • Development experience with GCP services (Cloud Storage, Cloud Spanner, BigQuery)
  • Experience with Apache Spark and NoSQL Implementation
  • Proficiency working with structured, semi-structured and unstructured data sets including social, web logs and real time streaming data feeds
  • Able to tune Big Data solutions to improve performance and end-user experience
  • Expert level usage with Jenkins, GitHub CI/CD is preferred
  • Must have the legal right to work in the United Kingdom