AI Architect

Job Summary Key Responsibilities • Design, build and maintain batch and real-time data pipelines. • Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL. • Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines • Optimise pipelines for scalability, reliability and performance. • Develop automated tests and participate in code reviews. • Contribute to CI/CD and engineering best practices. • Monitor, troubleshoot and support production workloads. • Use AI-assisted development tools to improve productivity while maintaining high engineering standards. Essential Skills • Databricks • Lakeflow Declarative Pipelines and DXQ, DB Genie, MCP Server • Apache Spark (PySpark) • Spark Structured Streaming • Auto Loader • Unity Catalog • Python • SQL • Delta Lake • Azure Event Hubs or equivalent event streaming platforms • Git and collaborative development workflows • Unit testing • CI/CD • Strong analytical and problem-solving skills Nice to Have • Performance tuning and Spark optimisation • Data modelling • Azure DevOps • AI coding assistants (GitHub Copilot, Cursor or similar) • Experience working in Agile or SAFe environments We're Looking For An Engineer Who • Takes ownership from design through production support. • Writes clean, maintainable and well-tested code. • Builds reusable, production-ready solutions. • Collaborates effectively with teammates and stakeholders. • Continuously learns and embraces modern engineering practices. Show more Show less

Design, build and maintain batch and real-time data pipelines. Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL. Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines. Optimise pipelines for scalability, reliability and performance. Develop automated tests and participate in code reviews. Contribute to CI/CD and engineering best practices. Monitor, troubleshoot and support production workloads. Use AI-assisted development tools to improve productivity while maintaining high engineering standards.