Lead Data Engineer

The Department of Agriculture, Fisheries and Forestry (DAFF) is looking for a Data Engineer to join its Enterprise Analytics and Technology Services (EATS) section to work across its data and analytics platforms. We are seeking candidates with strong experience in developing Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) processes, and /or the development of data products and complex data visualisations. The role will be responsible for design, development and testing activities across several data movement, data transformation, and data visualisation processes within DAFF. The data movement and transformation processes focus on the preparation of data for use in decision making processes across the department, utilising modern cloud technology (Azure & Databricks) to enable operational analytics use cases. To be successful in the role, you must have a strong work ethic including taking ownership, providing leadership, having a high level of productivity, and working with a multi-disciplinary team in an Agile development environment. You must have suitable qualifications or training in data engineering techniques and at least 5 years’ experience working as a data engineer. Candidates must hold a valid Negative Vetting Level 1 security clearance or be willing and eligible to obtain one. Commencement within the section is contingent on the candidate obtaining a Baseline clearance. Please provide your CSID on your resume/CV (if known). Key duties and responsibilities Responsibilities include, but not limited to: Be responsive, flexible, and work collaboratively as part of an agile team. Strong relationship building, and negotiation skills. Strong written and oral communication skills. Creating and maintaining automated ingest and transformation patterns and frameworks. Designing, building and maintaining data ingest and transformation solutions to meet current and emerging needs. Assisting project teams achieve objectives that align with departmental, divisional, and program priorities. Supporting data engineers in delivery teams, through regular quality reviews and constructive feedback on utilising data assets to produce quality data products. The successful candidate will require experience with the following techniques and technologies: Azure hosted services including: Data Integration - Data Factory, SQL Server Integration Services and/or Databricks Data Store - SQL Server and/or Data Lake Storage Analytics - Azure Databricks, Azure Machine Learning, ArcGIS Enterprise Development tools – DevOps, Visual Studio Data technology solutions – sourcing (Oracle, Ingres, Azure, SQL Server), automated ingestion Data Preparation Transformation of data into formats tailored for advanced analytics and AI use cases – Parquet and/or Delta Data Visualisation Analyse and interpret complex data sets, and to identify trends and patterns. Design principles to create appropriate visualisations for target audience. Visualisation tools - Power BI. Criteria The buyer has specified that each candidate must provide a response to each criterion. Each response is limited to 3000 characters. Essential criteria Weighting 1. Demonstrated experience developing ETL/ELT processes for complex and/or large data movement, transformation and/or visualisations, particularly in a cloud environment. Weighting: 50% 2. Experience preparing data optimised for query performance in cloud computed engines. E.g. • Distributed computing engines (Spark) • Graph Databases • Azure SQL Weighting: 25% 3. Experience working with Engineering, Storage and Analytics services in cloud infrastructure. Weighting: 25% Desirable criteria 1. Experience working with Azure Data Factory and Databricks. 2. Experience/Knowledge of working with Data Lake and Lakehouses. Opportunity summary Sellers can submit Up to 2 candidates Number of sellers invited More than 10 Number of candidates submitted Fewer than 10 Send us feedback About Accessibility Privacy Terms of use Disclaimer and copyright An initiative of the Digital Transformation Agency The Australian Government acknowledges the Traditional Owners of Country throughout Australia and acknowledges their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the Elders past and present. © Commonwealth of Australia