Senior Data Engineer

The Australian Electoral Commission (AEC) is seeking to engage a suitably qualified and experienced Data Engineer to deliver and support a range of ICT projects/programs. The Data Engineer will be required to work within multi-disciplinary project teams to design, develop, and optimise ETL and data pipelines producing analytical datasets embedded as part of data rich products supporting election systems. This role is pivotal in shaping the data architecture that underpins AEC’s analytics, forecasting and business intelligence capabilities leveraging the full power of the Microsoft ecosystem. The ideal candidate will be an experienced Data Engineer with expertise in implementing analytical solutions using Microsoft technologies working closely with stakeholders and data modellers to develop an end-to-end solution. They will be a self-starter with strong communication skills, a robust understanding of the data modelling and ETL lifecycle and have a bias towards collaborative problem solving. The candidate must hold an active NV1 security clearance prior to commencement. Key duties and responsibilities The Data Engineer will be responsible for, but not limited to: • Design and build ETL/ELT pipelines in Azure Data Factory and create/manage linked services, datasets, and integration runtimes. • Write and optimize complex T-SQL (stored procedures, functions, views) and performance tuning, query optimization, execution plans, indexing strategy • Manage source control with Git (branching strategy, pull requests) and maintain Azure DevOps pipelines for automated deployment of ADF resources and infrastructure across Dev . Test . Prod environments. • Work across Azure services: Data Lake Storage (ADLS Gen2), Blob Storage, Azure SQL, Synapse, Key Vault, Azure Functions, Logic Apps. • Monitor pipeline runs, set up alerts (Azure Monitor, Log Analytics) and troubleshoot failures and analyse logs, metrics, and monitoring dashboards to detect anomalies and failures early. • Collaborate with data architects, data modellers, analysts, and business stakeholders to understand data requirements and translate them into technical solutions. • Following the Scaled Agile Framework (SAFe) methodology for software development and contributing to Agile ceremonies. • Production Support: Monitor, maintain, and support data integration workflows, pipelines, and ETL processes using Azure Data Factory. • Incident Management: Respond to and resolve incidents, service requests, and system alerts related to data platform operations in a timely manner. • Problem Resolution: Investigate, troubleshoot, and remediate data and infrastructure issues, working closely with development and infrastructure teams as needed. • Ability to participate in on-call support rotations and respond to high-priority incidents as needed. • Excellent analytical, problem-solving, and communication skills • Adhering to coding best practices and AEC code quality standards including version control, code documentation, and testing. Criteria The buyer has specified that each candidate must provide a one page pitch to address all criteria specified. This is equal to 5000 characters. Essential criteria 1. • Demonstrated experience designing, developing, and orchestrating scalable ETL/ELT pipelines using Azure Data Factory — including parameterized, metadata-driven pipelines, integration runtimes, incremental loads, and robust error handling to move and transform data from diverse sources. • Advanced ability to write, optimize, and troubleshoot complex T-SQL (stored procedures, functions, views), design dimensional data models (star/snowflake schemas), and perform performance tuning through indexing and query optimization. • Practical experience ingesting and exchanging data via REST/SOAP APIs handling authentication (OAuth2, tokens, managed identities), pagination, rate limiting, and parsing JSON/XML responses into structured, analytics-ready datasets. • Exceptional communication and documentation abilities, with a proven track record of collaborating across technical and business teams. • Solid understanding of the wider Azure ecosystem (ADLS Gen2, Azure SQL, Key Vault, Azure Functions, Synapse) combined with strong analytical and troubleshooting skills to monitor pipelines, diagnose failures, and apply security best practices. 2. To be considered compliant, submissions need to be provided in 1 PDF Document including: Page 1 - a completed copy of the AEC Candidate Submission Guide provided as part of the industry brief (Emailed to the listed BuyICT contact address upon RFQ publication): • Provide information for the following items: Candidate Name, Position Title, Opportunity ID, Candidate Availability, Location, Citizenship Status, Security Clearance Level, Planned Leave (exceeding 1 week), Additional Engagements, Submitting Consultant. • Choose an item or check the box that applies to the following options: Previous AEC Employee or Contractor, Current AEC Contractor, Code of Conduct, Submission History, Authority to Represent, Submission Compliancy. • Consultant comments may be added (must be limited to the space provided). Page 2 - A one-page pitch addressing the selection criteria. This should not exceed one page. Pages 3 - 5 - A resume of no more than three pages. We will strictly adhere to the above requirements. Submissions will be deemed non-compliant if they do not meet the listed criteria, if they exceed the 5-page limit or if the rates of submitted candidates are included in the PDF. Desirable criteria 1. • Background in building and managing CI/CD pipelines using Azure DevOps, enabling streamlined deployment and operational efficiency. • Prior experience in federal government data domains. • Bachelor's or Master's degree in a relevant field (e.g. Computer Science, Data Science, Statistics, Engineering). Opportunity summary Sellers can submit Up to 2 candidates Number of sellers invited Between 5 and 10 Number of candidates submitted Fewer than 10 RFQ closing time 01 Weeks 01 Days 00 Hours Send us feedback About Accessibility Privacy Terms of use Disclaimer and copyright An initiative of the Digital Transformation Agency The Australian Government acknowledges the Traditional Owners of Country throughout Australia and acknowledges their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the Elders past and present. © Commonwealth of Australia