About Blackbook AI
Blackbook AI is a technology consultancy working across AI, automation, data and digital. We started in Australia and our head office is still in Brisbane, but we've grown into APAC and North America over the past nine years, and that now includes a Calgary team we're actively building out. In that time we've delivered over 2,000 projects for more than 180 clients.
We're a Databricks Consulting Partner, along with being a Microsoft Solutions Partner, an AWS Consulting Partner and a UiPath Diamond Partner. We hire locally because people who understand the market tend to do better work in it, and we back them with a global delivery team behind the scenes.
The role
We're looking for a Databricks Data Engineer to join our Calgary team and help clients modernise their data platforms, from the first conversation through to production.
Here's the part that matters most. Plenty of engineers can build a pipeline once someone hands them a spec. What sets our team apart is the people who can get in a room with a GM, a finance lead and a plant supervisor who all describe the same problem differently, work out what's actually going on, and walk out with a clear picture of what needs building and why.
That means this role is genuinely two jobs in one. You need the technical depth to design and build lakehouse solutions on Databricks that hold up under load. You also need the business analysis and consulting skills to run the discovery, ask the uncomfortable questions, challenge a request that doesn't make sense, and bring stakeholders along with you. We're not looking for someone who waits for requirements. We're looking for someone who goes and finds them.
You'll see projects through end to end rather than picking up someone else's backlog, and you'll work across a range of industries including energy, utilities, transport, financial services and manufacturing.
What you'll be doing
Working with the business
- Running discovery sessions and requirements workshops with stakeholders who often can't articulate what they need yet
- Getting past the stated request to the underlying problem, and being willing to push back when the two don't line up
- Mapping current processes, data flows and pain points, then documenting requirements and acceptance criteria clearly enough that everyone agrees before a line of code gets written
- Translating in both directions, so business problems into technical design, and technical constraints into language a non-technical exec can act on
- Managing scope and expectations across an engagement, including saying "not yet" when that's the right answer
- Presenting designs, trade-offs and progress to audiences ranging from data teams to executive sponsors
- Building the kind of client relationships that turn into repeat work, and spotting where we could help further
Building the solution
- Designing, building and optimising data pipelines on Databricks using PySpark, Spark SQL and Python
- Implementing lakehouse architectures with Delta Lake, Unity Catalog, medallion patterns and Databricks Workflows
- Migrating legacy warehouses, older ETL tooling and on-premise workloads across to Databricks
- Building ingestion from whatever the client has: APIs, streaming sources, relational databases, flat files, SaaS platforms
- Bringing proper engineering discipline to data work, so version control, CI/CD, automated testing and infrastructure as code
- Setting up data governance, lineage, access control and quality monitoring
- Tuning performance and keeping an eye on cost across clusters, jobs and SQL warehouses
- Mentoring junior engineers and having a say in how our data practice runs
What we're looking for
On the consulting and business analysis side
- Real experience running workshops and requirements sessions, not just attending them
- A track record of working directly with business stakeholders rather than through a project manager or BA
- Strong elicitation instincts, so knowing which questions to ask and when to keep digging
- Clear written communication, including requirements documentation, process maps and design docs that people actually read
- Confidence presenting to senior stakeholders and holding your position when you're challenged
- Commercial awareness, so you can tell the difference between what a client asked for and what will actually deliver value
- Comfort with ambiguity, because early-stage engagements rarely arrive with clean requirements
On the technical side
- 4+ years in data engineering, with at least 2 of those hands-on with Databricks
- Strong Python and advanced SQL
- Solid PySpark and distributed processing experience, including being able to troubleshoot and optimise Spark jobs
- Production experience with Delta Lake and medallion (bronze, silver, gold) architecture
- Cloud experience on Azure (ADLS Gen2, Data Factory, Azure DevOps) or AWS (S3, Glue, Lambda)
- Comfortable with Git-based workflows and CI/CD for data pipelines
- Legally entitled to work in Canada
You don't need a formal BA qualification. We care about whether you can do the work, not whether you've got the certificate for it.
Nice to have
- Databricks certification, either Data Engineer Associate or Professional
- Unity Catalog rollouts and data governance work
- Lakeflow Declarative Pipelines / Delta Live Tables, structured streaming, or Databricks Asset Bundles
- Terraform or similar infrastructure-as-code tooling
- dbt or Airflow
- Power BI, Tableau or equivalent
- MLflow, or supporting ML and GenAI workloads on Databricks
- Prior consulting or professional services experience
- Presales or solution estimation exposure
- Energy, utilities or resources experience, which is where a lot of Calgary's interesting data sits
Why you'd want to work here
You'll get real variety. Multiple clients, multiple industries, and no two engagements that look the same.
You'll also stay close to the business rather than being kept at arm's length from it. Our engineers sit in the discovery sessions, not just the sprint reviews, and that's deliberate. It's why our solutions land.
You'll get the certifications and training to go with it. Our Databricks, Microsoft, AWS and UiPath partnerships mean we invest in keeping people current, not just billable.
And because the Canadian practice is still growing, you'll have a genuine say in how it takes shape.
How to apply
Apply through Indeed with your resume. We'd also like to hear about a time you changed what a client was asking for, so where you dug into a request, found the real problem underneath it, and delivered something different to what was originally scoped. A short paragraph is plenty.
Applicants must be legally entitled to work in Canada. We're not able to provide visa sponsorship for this role.
Pay: $100,000.00-$160,000.00 per year
Work Location: In person
