Data Engineer

The Role
As a Data Engineer, you will be responsible for building and maintaining data pipelines. We have multiple database paradigms, from relational databases to document databases. You will need to build solutions to stream and/or batch data changes into the data warehouse (Snowflake).
Your solutions should give engineers the freedom to build applications without worrying about data modelling, warehousing or analysis. They will continue to develop applications and make changes to the database, and you are to ensure that all the modifications are accurately captured in the data warehouse. You will also need to assist data analysts to build transformation pipelines using tools such as dbt to enable them to remodel and transform datasets for analysis.
You will be working with software engineers, data scientists and analysts to ensure that data is continuously copied across different databases and the data remains fresh.
Reporting to: The Head of Engineering.
Required Skills
  • Knowledge of cloud computing is required (AWS, GCP or Azure)
  • Experience with ELT / ELT tools. (e.g. Matillion, Azure Data Factory)
  • Experience with data warehouses (e.g. Snowflake, AWS Redshift, Azure Synapse)
  • Strong Python & SQL skills.
Preferred Skills
  • Experience with data orchestration tools (e.g. Airflow, Dagster, Prefect).
  • Experience with data testing & data transformation tools  (e.g. Great Expectations, dbt)
  • Experience with code as IaC is plus. (eg. CDK, Terraform)
What you will be doing
  • Work with the engineering team to develop data pipelines to stream data from application databases to data warehouses. 
  • Ensure that scheduled tasks are completed on time, and all ELT operations are captured accurately.
  • Work with the engineering team to ensure that DML and DDL operations are captured quickly and accurately to ensure that the data in the data warehouse remains fresh.
  • Build monitoring solutions to ensure that data pipelines are not delayed.

APPLY NOW

Acknowledgement of Country

U Group acknowledges the deep connections of Aboriginal and Torres Strait Islander communities to Country. We pay our respects to the Whadjuk Nyoongar people, who are the Traditional Owners of the land our offices are on in Perth, Western Australia