Senior Data Engineer

Data Engineers prepare raw data for analytical or operational uses. They make data accessible so that it can be used for optimisation, delivery of services and enhanced analytics solutions. Data Engineers design, build and maintain data products, services and platforms that support the agency’s operational, analytical and business intelligence needs. They transform and manage data throughout the data lifecycle to ensure data assets are secure, reliable, accessible and fit for purpose. Data Engineers work collaboratively with business and technical stakeholders to deliver scalable data solutions, support evidence-based decision making, and contribute to the continuous improvement of the agency’s data ecosystem. Senior Data Engineers apply technical expertise and judgement to design and develop integrated data solutions that address business and operational requirements. They translate requirements into practical outcomes, improve the effectiveness, reliability and maturity of data engineering practices, and resolve technical challenges. Senior Data Engineers contribute to the development of standards, innovative approaches and improved ways of working, while providing technical guidance and supporting knowledge sharing across teams. Key duties and responsibilities Duties may include some or all of the following: Engage with stakeholders to define business requirements and shape data, analytics and software solutions that meet agency needs. Design, develop and maintain data engineering, enterprise data warehouse, data lake, streaming and software solutions. Build, maintain and support high-quality code, technical designs, prototypes and complex development solutions across the product lifecycle. Perform complex data extraction, manipulation, analysis, querying, programming, testing and troubleshooting to support effective business outcomes. Provide technical advice, documentation input, go-live support and ongoing maintenance for complex ICT and development activities. Apply endorsed methodologies, standards, security measures, risk controls and best practices to improve availability, performance, compliance and reliability of ICT solutions. Technical Skills include: Data modelling techniques (relational, dimensional, messaging and big data), data analysis and profiling using modern analytical tools and programming languages, and visualisation methods and tools. Experience with one or more of the following - Teradata, Cloudera Data Platform. Experience in implementing solutions on Red Hat Open shift platforms is highly desirable. Best-practice data cleansing, validation, quality assurance and feature preparation techniques for complex datasets. Programming & Software Engineering – one or more of following: Modern engineering practices supporting data products (API integration, Data Replication, Data Acquisition, Data Reconciliation, ETL/ELT using Batch and near Real Time) Programming in one or more of the following - Python, SQL, Apache Spark, Java, Apache NiFi. Experience in utilising Source control (Git) repositories, utilising and building CI/CD pipelines Automated testing, Secure coding practices, Code review processes Criteria The buyer has specified that each candidate must provide a one page pitch to address all criteria specified. This is equal to 5000 characters. Essential criteria 1. Demonstrated experience engaging stakeholders to define business requirements and design, develop, implement, maintain and support data engineering, enterprise data warehouse, data lake, streaming and software solutions that meet complex agency needs. 2. Demonstrated ability to perform complex data extraction, manipulation, analysis, querying, programming, testing and troubleshooting, while applying data quality, integrity, security, risk controls and endorsed engineering standards to deliver reliable business outcomes. 3. Demonstrated technical capability across data modelling, analysis, profiling, cleansing, validation, API integration, data replication, data acquisition, data reconciliation, ETL/ELT, batch and near real-time pipeline delivery. Desirable criteria 1. Experience with one or more of the following platforms: Teradata, Cloudera Data Platform, Red Hat OpenShift 2. Experience in one or more of the following: Python, SQL, Apache Spark, Java, Apache NiFi, Hadoop 3. Experience building and supporting data products using automated testing, secure coding practices, Git repositories, DevOps pipelines, code review processes and contemporary software engineering practices. 4. Relevant qualification or equivalent practical experience in computer science, software engineering, data engineering, enterprise data warehousing or software development. Opportunity summary Sellers can submit Up to 4 candidates Number of sellers invited More than 10 Number of candidates submitted Fewer than 10 RFQ closing time 01 Weeks 06 Days 12 Hours Send us feedback About Accessibility Privacy Terms of use Disclaimer and copyright An initiative of the Digital Transformation Agency The Australian Government acknowledges the Traditional Owners of Country throughout Australia and acknowledges their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the Elders past and present. © Commonwealth of Australia