Lead Data Engineer

The Data Engineer is responsible for designing, developing, implementing and supporting enterprise data solutions that enable the secure, reliable and efficient collection, integration, transformation and delivery of data across the organisation. The role works with modern cloud-based data platforms to develop scalable data pipelines, data models and analytical solutions that support reporting, business intelligence, data analytics, artificial intelligence and operational decision-making. Key duties and responsibilities Data Engineer (Databricks): The Data Engineer is responsible for designing, developing and supporting enterprise data solutions that enable the secure, reliable and scalable collection, integration, transformation and delivery of data across the organisation. The role develops and maintains modern data platforms, data pipelines and lakehouse architectures using Databricks and associated cloud technologies to support analytics, business intelligence, machine learning and artificial intelligence initiatives. Design, develop and maintain enterprise-scale data pipelines and ELT/ETL solutions using Databricks.Build and optimise Lakehouse architectures using Delta Lake and Medallion design patterns.Develop data ingestion frameworks for structured, semi-structured and unstructured data sources.Create scalable data transformation processes using PySpark, Spark SQL and Python.Implement data quality, lineage, monitoring and observability capabilities.Design and maintain dimensional and analytical data models to support reporting and advanced analytics.Support machine learning and AI workloads by providing curated and trusted datasets.Implement CI/CD, Infrastructure as Code and automated testing practices across data engineering solutions.Monitor platform performance, troubleshoot production issues and implement continuous improvements.Ensure solutions comply with APS information security, privacy, governance and record-keeping requirements.Provide technical leadership, mentoring and knowledge sharing across project and operational teams.Engage with stakeholders to translate business requirements into scalable and sustainable data solutions. Technical: Significant experience designing and implementing solutions on the Databricks Data Intelligence Platform.Advanced proficiency in PySpark, Spark SQL, Python and SQL.Experience with Databricks Workflows, Delta Lake, Unity Catalog and Databricks SQL.Strong understanding of Lakehouse architecture, Medallion data models and modern data engineering patterns.Experience integrating data from APIs, databases, SaaS platforms and enterprise systems.Knowledge of cloud platforms such as Microsoft Azure, including ADLS Gen2, Azure Data Factory and Microsoft Fabric.Experience implementing DevOps practices using Git, CI/CD pipelines and automated testing.Understanding of data governance, metadata management, data quality frameworks and information security controls.Experience supporting analytics, AI and machine learning workloads within enterprise environments. Data Engineer (Microsoft Fabric): The Data Engineer designs, develops and supports enterprise data solutions using Microsoft Fabric to enable analytics, reporting and data-driven decision-making. The role applies data engineering practices to deliver scalable, secure and governed data capabilities that improve information accessibility, operational efficiency and organisational data maturity. Design and implement data pipelines, lakehouse solutions and data products using Microsoft Fabric.Develop and maintain enterprise data integration and transformation capabilities.Support Fabric components including Data Factory, Data Engineering, Lakehouse, Warehouse and Real-Time Analytics workloads.Develop scalable data models and datasets supporting analytics and reporting requirements.Apply testing, automation, source control and CI/CD practices across data solutions.Ensure compliance with data governance, quality, privacy and security requirements.Monitor, support and optimise platform performance and service reliability.Collaborate with stakeholders to define and deliver business-focused data solutions.Maintain technical documentation, operational runbooks and data lineage artefacts.Contribute to platform roadmap, innovation and continuous improvement activities. Technical: An ideal candidate would have 2-5+ years of data engineering experience, strong SQL and Python skills, practical experience with Microsoft Fabric, exposure to lakehouse and warehouse architectures, familiarity with Azure DevOps and CI/CD, and an understanding of government data governance, privacy and security requirements. They should be capable of independently designing, building, deploying and supporting enterprise-scale data solutions while working closely with stakeholders.