Senior Data Engineer
Newcastle upon Tyne, Tyne and Wear
22 Aug 2019
- Lead the design, implementation, and continuous delivery of pipelines using distributed Azure based big data technologies supporting data processing initiatives across batch and streaming datasets
- Responsible for development using Scala, Python languages and Big Data Frameworks such as Spark, EMR, Kafka, Storm, Jenkins, Jfrog Artifactory and DataBricks
- Provide oversight and direction on deployed Azure platform components
- Process and manage high volume real time customer interaction streams
- Support the data analytic team in the creation of data models and product/propositions.
- Development and implementation of data flows and ETL's to ingest data, using both streaming and batch technologies from multiple sources
- Design and development of the deployment processes using continuous integration
- Provide technical review of the solution and identify project risks and estimate development time for such solution.
- Support entire development process and conduct code reviews.
- Bachelor's Degree or higher in Computer Sciences or similar
- Development experience with Azure services
- Development experience with GCP services (Cloud Storage, Cloud Spanner, BigQuery)
- Experience with Apache Spark and NoSQL Implementation
- Proficiency working with structured, semi-structured and unstructured data sets including social, web logs and real time streaming data feeds
- Able to tune Big Data solutions to improve performance and end-user experience
- Expert level usage with Jenkins, GitHub CI/CD is preferred
- Must have the legal right to work in the United Kingdom