Senior Data Engineer

Data Engineers prepare raw data for analytical or operational uses. They make data accessible so that it can be used for optimisation, delivery of services and enhanced analytics solutions. Key duties and responsibilities Role Overview The ALA is a National Collaborative Research Infrastructure Strategy (NCRIS) research infrastructure hosted by CSIRO, responsible for mobilising biodiversity data to support national and international users who demand timely access to Australian biodiversity data. The ALA annually delivers biodiversity data to over 30,000 users in research, industry, and government. As the Australian node of the Global Biodiversity Information Facility (GBIF), the ALA ensures that Australian biodiversity is available to support the international science community and provides reciprocal data benefits to Australian users.   Joining the ALA’s nationally distributed Data Team you will work on data acquisition, transformation, loading and quality assurance on a large-scale cloud-based system, building and operating automated data harvesting, ingestion and indexing jobs. The role requires strong collaboration and communication skills to act as a reliable point of contact for data providers and to be part of a strong team culture where the majority of tasks require the input and expertise of others. The Data Team’s work has reach across many applications and stakeholders of the ALA. Data engineering Build and manage automated and manual data loading processes Map datasets to the Darwin Core standard Implement, deploy, schedule, monitor and maintain data load and indexing processes Implement quality assurance and verification on datasets to ensure loaded records meet expectation Stakeholder engagement Engage professionally with external stakeholders offering technical guidance on data management issues such as data mapping, automation, and loading Present to external stakeholder groups on ALA data loading processes Respond to ALA support queries as required CSIRO Values Communicate effectively and respectfully with all staff, clients and suppliers in the interests of good business practice, collaboration and enhancement of CSIRO’s reputation. Work collaboratively with colleagues within your team, the broader CSIRO and across partner institutions to reach objectives. Adhere to the spirit and practice of CSIRO’s Values, Health, Safety and Environment plans and policies, Diversity initiatives and Zero Harm goals.   Other Duties Complete other duties as directed Criteria The buyer has specified that each candidate must provide a one page pitch to address all criteria specified. This is equal to 5000 characters. Essential criteria 1. • 4+ years demonstrated operations experience in a data driven production environment • Strong knowledge of scripting languages – Python, Spark, Scala, SQL, bash, R, JavaScript; strong ETL skills with large datasets with a focus on efficiency and scale • Experience with Linux OS and cloud computing specifically Airflow and AWS EMR and experience in both delivering and consuming REST services • Source code management using git, svn or Bitbucket • Knowledge of SOLR and/or Elasticsearch administration and queries Desirable criteria 1. • Experience with a variety of open source relational and non-relational databases • Effective stakeholder engagement and technical liaison skills Opportunity summary Sellers can submit Up to 1 candidate Number of sellers invited Fewer than 5 Number of candidates submitted Fewer than 10 RFQ closing time 01 Weeks 06 Days 12 Hours Send us feedback About Accessibility Privacy Terms of use Disclaimer and copyright An initiative of the Digital Transformation Agency The Australian Government acknowledges the Traditional Owners of Country throughout Australia and acknowledges their continuing connection to land, waters and community. We pay our respects to the people, the cultures and the Elders past and present. © Commonwealth of Australia