Senior Data Engineer

The Australian Electoral Commission (AEC) is seeking to engage a suitably qualified and experienced Data Engineer to deliver and support a range of ICT projects/programs. The Data Engineer will be required to work within multi-disciplinary project teams to design, develop, and optimise ETL and data pipelines producing analytical datasets embedded as part of data rich products supporting election systems. This role is pivotal in shaping the data architecture that underpins AEC’s analytics, forecasting and business intelligence capabilities leveraging the full power of the Microsoft ecosystem. The ideal candidate will be an experienced Data Engineer with expertise in implementing analytical solutions using Microsoft technologies working closely with stakeholders and data modellers to develop an end-to-end solution.They will be a self-starter with strong communication skills, a robust understanding of the data modelling and ETL lifecycle and have a bias towards collaborative problem solving.   The candidate must hold an active NV1 security clearance prior to commencement. Key duties and responsibilities The Data Engineer will be responsible for, but not limited to: • Design and build ETL/ELT pipelines in Azure Data Factory and create/manage linked services, datasets, and integration runtimes. • Write and optimize complex T-SQL (stored procedures, functions, views) and performance tuning, query optimization, execution plans, indexing strategy • Manage source control with Git (branching strategy, pull requests) and maintain Azure DevOps pipelines for automated deployment of ADF resources and infrastructure across Dev . Test . Prod environments. • Work across Azure services: Data Lake Storage (ADLS Gen2), Blob Storage, Azure SQL, Synapse, Key Vault, Azure Functions, Logic Apps. • Monitor pipeline runs, set up alerts (Azure Monitor, Log Analytics) and troubleshoot failures and analyse logs, metrics, and monitoring dashboards to detect anomalies and failures early. • Collaborate with data architects, data modellers, analysts, and business stakeholders to understand data requirements and translate them into technical solutions. • Following the Scaled Agile Framework (SAFe) methodology for software development and contributing to Agile ceremonies. • Production Support: Monitor, maintain, and support data integration workflows, pipelines, and ETL processes using Azure Data Factory. • Incident Management: Respond to and resolve incidents, service requests, and system alerts related to data platform operations in a timely manner. • Problem Resolution: Investigate, troubleshoot, and remediate data and infrastructure issues, working closely with development and infrastructure teams as needed. • Ability to participate in on-call support rotations and respond to high-priority incidents as needed. • Excellent analytical, problem-solving, and communication skills • Adhering to coding best practices and AEC code quality standards including version control, code documentation, and testing.