Big Data Engineer (Azure), G1

MBN are partnering with a global business founded and headquartered in Glasgow to help them continue their evolution within Data Engineering. After the creation and development of a brand new Azure big data platform, the business are looking to expand by bringing in a new member of the team to work on a variety of cutting edge big data projects. Key Responsibilities: * Design and implement pipelines to ingest data into and consume data from the data lake * Source and transform data from historical and streaming data sources * Implement automated monitoring and alerting on pipelines to ensure consistent processing of data * Design systems to allow real time analytics on high velocity data * Proactively ensure optimum performance by revisiting design choices to check they are still fit for purpose * Collaborate with Data Scientists and BI Developers to ensure the wider team?s objectives are being met by data lake design * Support wider team in consuming data from the data lake * Be able to provide technical leadership in the use and design of big data technologies Key Skills & Experience Required: * Experience ingesting and streaming data in to a cloud environment * Experience creating robust well engineered pipelines (ideally using Python or PySpark) * Experience working in a big data environment or cloud platform (Azure being preferential but not essential) * Working knowledge of the Azure Stack is preferred using tools such as (CosmosDB, Data Factory, Logic Apps, Event Hub / IOT Hub, DataBricks) Highly advantageous