Senior Databricks, Fabric Data Engineer

Data Engineers prepare raw data for analytical or operational uses. They make data accessible so that it can be used for optimisation, delivery of services and enhanced analytics solutions.  The Data, Reporting and Systems Improvement Specialist will provide hands-on technical support across EPA’s data and reporting environment. Working within the Data and Knowledge Management section, the role will develop and maintain data pipelines, datasets and reporting products; address data quality and integration issues; strengthen technical documentation; and improve the resilience of critical Databricks and Power BI functions. The specialist will work collaboratively with APS staff, business areas and ICT specialists, with a strong focus on knowledge transfer and reducing key-person dependency. Key duties and responsibilities Develop, maintain and optimise data pipelines, transformations and curated datasets in Databricks. Investigate and resolve data quality, integration, refresh and reporting issues. Design and maintain fit-for-purpose data models and support Power BI datasets and reports. Document data assets, technical processes, dependencies, controls and support procedures. Contribute to priority data, reporting and systems-improvement initiatives. Apply security, privacy, data-governance and change-control requirements. Provide technical advice to DKM, business stakeholders and ICT specialists. Transfer knowledge to APS staff and support operational continuity. Criteria The buyer has specified that each candidate must provide a one page pitch to address all criteria specified. This is equal to 5000 characters. Essential criteria 1. Data access: Level 1 (DCF) Can use the range of available options to access common data sources 2. Data editing: Level 2 (DCF) Has a comprehensive knowledge of the different editing methods at their disposal Understands why different methods are used and can describe the limitations of each method Knows where to find expert advice about data editing as required 3. Data integrity and quality assurance: Level 2 (DCF) Can describe and produce data quality measures for the outputs they produce Has a comprehensive knowledge of relevant data quality measures and can use them to make accurate assessments of data fitness-for purpose (including trustworthiness and accuracy) Has a comprehensive understanding of relevant data integrity principles and practices, and uses them to maintain the integrity of organisational data 4. Data use and re-use: Level 2 (DCF) Understands and can articulate the value of data in terms of use and re-use Implements a variety of techniques to ensure data is open and can be used beyond the specific purpose for which it was collected, in accordance with legislative requirements and organisational guidelines (includes data sharing legislation) 5. Improvement and Innovation - Data processes/systems and tools/products: Level 2 (DCF) Can identify deficiencies in current processes/systems and tools/ products, gain the required approval to make changes, and lead the implementation of those changes Can identify and harness opportunities to create efficiencies and effective new processes/systems and tools/products Can develop a test scenario and lead systems testing, following organisational procedures and protocols and analyse the results 6. Integrate data: Level 2 (DCF) Can perform data integration using standard tools and can implement quality controls Knows where to obtain expert advice on data integration as needed Is able to understand and amend existing code (programming language) to integrate data Can assess datasets' suitability for linkage and extract appropriately prepared datasets 7. Value organisational data as assets: Level 2 (DCF) Has extensive knowledge of the organisation's data assets, including a comprehensive understanding of how their fitness for purpose translates to value for the organisation Promotes opportunities for using data to support decision making, advice and research Considers the broader environment and context when drawing or interpreting insights from data. Uses these insights to make informed, evidence-based decisions and recommendations, and act accordingly Opportunity summary Sellers can submit Up to 6 candidates Number of sellers invited Between 5 and 10 Number of candidates submitted Fewer than 10 RFQ closing time 01 Weeks 04 Days 06 Hours Send us feedback About Accessibility Privacy Terms of use Disclaimer and copyright An initiative of the Digital Transformation Agency The Australian Government acknowledges the Traditional Owners of Country throughout Australia and acknowledges their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the Elders past and present. © Commonwealth of Australia