Senior Data Engineer

Role/s: 1 X APS6 Data Engineer The APS6 Data Engineer (Labour Hire) will design, build and maintain reliable data pipelines and data products that enable the Data Science Hub's data science, machine learning, artificial intelligence and advanced analytics work. The role will help move solutions from exploration and prototype through to repeatable, secure and supportable delivery.  Working closely with data scientists, machine learning engineers, solution architects, platform teams and business stakeholders, the role will turn data requirements into well-engineered datasets and services. The Data Engineer will apply strong software engineering, cloud and DevOps practices while supporting data quality, lineage, security, privacy, monitoring and responsible use of Agency data.  Supplier Briefing: A supplier briefing will be held for this role on TBC – meeting details will be posted in the “Q&A” section of the RFQ for invited Sellers only. Rate: The proposed rate should reflect candidate's skills and experience against the Evaluation Criteria. Candidates that do not demonstrate these criteria will not be assessed. Candidates should be aware that rates may be negotiated further as part of the selection process. Citizenship: As part of the eligibility and suitability requirement, NDIA seeks Labour Hire Workers who are Australian citizens only. Successful candidates will be required to furnish valid evidence of citizenship during the Pre-engagement Check. Labour Hire Licence: Applicable for ACT, VIC and QLD: Labour hire licences are required in the state that specified personnel are being contracted.   Ability to obtain a Baseline clearance (NV1 preferable) Key duties and responsibilities Responsibilities of the role include but are not limited to: Design, build, test and maintain scalable batch and event-driven data pipelines for analytics, machine learning and AI use cases.  Develop curated, reusable and well-documented datasets and data products from structured and unstructured sources.  Work with data scientists and machine learning engineers to productionise data preparation, feature engineering, model input and model output pipelines.  Translate business and analytical requirements into technical designs, delivery estimates, work items and acceptance criteria.  Implement automated data quality controls, validation, metadata, lineage, logging, monitoring and alerting.  Optimise SQL, Python and cloud workloads for performance, reliability, security and cost efficiency.  Apply infrastructure-as-code, version control, automated testing, CI/CD and environment management practices to data engineering solutions.  Prepare and maintain technical documentation, including data mappings, interface specifications, runbooks, support procedures and architectural decisions.  Collaborate with Agency data custodians, cyber security, privacy, platform, architecture and delivery teams to ensure solutions meet governance and operational requirements.  (NOTE: the key responsibilities of the role are based on current priorities and may change over time Criteria The buyer has specified that each candidate must provide a one page pitch to address all criteria specified. This is equal to 5000 characters. Essential criteria 1. Demonstrated experience designing and delivering production-quality data pipelines and data products in AWS, using services such as S3, Glue, Lambda, Athena, EMR, Kinesis, Step Functions or equivalent managed services. 2. Strong experience with ETL/ELT patterns, data modelling, orchestration, data quality testing, metadata and lineage across structured and unstructured data. 3. Demonstrated application of modern software engineering and DevOps practices, including Git, code review, automated testing, CI/CD and infrastructure-as-code using Terraform or AWS CloudFormation. 4. Strong programming capability in Python, with experience developing maintainable, modular and testable production code. Experience with Java or Scala is also relevant. 5. Advanced SQL skills, including query optimisation, performance tuning and working with large or complex datasets. 6. Hands-on experience delivering cloud-based data solutions in containerised environments such as Docker, ECS or EKS, with a sound understanding of security, access controls, secrets management and operational monitoring. Desirable criteria 1. Experience supporting data science, machine learning, generative AI or document intelligence solutions, including data preparation, feature pipelines, vector or search-based retrieval, model outputs and evaluation datasets. 2. Experience with one or more modern data platforms or processing technologies such as Snowflake, Databricks, Redshift, Spark or dbt. 3. Knowledge of MLOps or DataOps practices, including experiment-to-production handover, reproducibility, model and data versioning, monitoring and automated deployment. 4. SAS Base Programming Opportunity summary Sellers can submit Up to 3 candidates Number of sellers invited More than 10 Number of candidates submitted Fewer than 10 RFQ closing time 01 Weeks 03 Days 12 Hours Send us feedback About Accessibility Privacy Terms of use Disclaimer and copyright An initiative of the Digital Transformation Agency The Australian Government acknowledges the Traditional Owners of Country throughout Australia and acknowledges their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the Elders past and present. © Commonwealth of Australia