Responsibilities:
- ETL Development & Optimization:
- Design, develop, and maintain ETL processes in Databricks for data warehousing, data lakes, and analytics, utilizing tools like Delta Lake and Delta Live Tables.
- Work with data architects and business teams for efficient data transformation and movement, managing both structured and unstructured data, including Change Data Capture (CDC) and streaming data.
- Data Design & Management:
- Develop primary keys, indexing strategies, and schemas to enhance performance and data integrity.
- Model and map data sources to targets, implement schema validation rules, and ensure data quality through validation checks.
- ETL Strategy & Pipeline Development:
- Create design documents and test cases, manage error-handling and logging mechanisms, and develop pipelines for incremental and full data loads.
- Performance & Troubleshooting:
- Optimize ETL performance, troubleshoot pipeline issues, and maintain data consistency checks. Develop production deployment steps, monitor ETL processes post-go-live, and recommend improvements.
- Documentation & Knowledge Transfer:
- Document ETL design, troubleshooting steps, and configuration; share knowledge with Ministry staff and ensure all processes are clearly documented.
Skills and Experience:
- ETL & Database Management:
- 7+ years of experience with ETL tools (e.g., Microsoft SSIS, T-SQL) and 2+ years with Delta Lake, Databricks, Azure Databricks, and Python/PySpark.
- Expertise in Delta Lake, Databricks Workflows, and Medallion Architecture for data quality.
- Technical Knowledge:
- Strong knowledge of SQL Server and Oracle, with skills in SQL scripting, data modeling, and building data ingestion/CDC processes using Oracle GoldenGate.
- Proficiency in Databricks, Delta Lake, Delta Live Tables, Spark, and cloud storage integrations with platforms like Azure, AWS, or Google Cloud.
- Documentation & Design Analysis:
- Experience with schema definitions, error handling, logging, data validation checks, and security policies within ETL.
- Strong analytical skills for troubleshooting and Fit-Gap analysis.
- Certifications (Preferred):
- Databricks Certified Data Engineer Associate or Professional, Microsoft Azure Data Engineer Associate, AWS Data Analytics Specialty, or Google Cloud Professional Data Engineer.
Job Types: Full-time, Fixed term contract
Pay: $80.00-$100.00 per hour
Experience:
- ETL tools (e.g., Microsoft SSIS, T-SQL): 7 years (required)
- Delta Lake, Databricks, Azure Databricks, and Python/PySpark: 2 years (required)
- SQL Server, Oracle, SQL scripting, data modeling, GoldenGate: 2 years (required)
Work Location: Hybrid remote in Toronto, ON M5V 1R5
