Job Description
Team: IT
This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Software Engineer, AI in Canada.
This role sits at the core of next-generation AI system development, focusing on how large language model-powered products are evaluated, improved, and deployed at scale. You will design and build robust AI evaluation frameworks and production-grade agent architectures that power intelligent features across complex product surfaces. Working in a highly technical, collaborative engineering environment, you will help define how AI performance is measured, trusted, and continuously optimized. The role blends hands-on systems engineering with deep AI/ML thinking, including LLM orchestration, retrieval systems, and observability. You will contribute to production AI systems while shaping the methodologies that determine model quality and reliability. This is a high-impact position where your work directly influences how AI behaves in real-world, user-facing applications.
Accountabilities:
- Design and implement end-to-end AI evaluation frameworks, including offline evaluation systems, production tracing, and human-in-the-loop feedback loops.
- Define and operationalize key AI performance metrics such as task success rate, hallucination detection, response quality, and business impact indicators.
- Build and maintain evaluation datasets, automated test harnesses, and regression detection pipelines for AI systems.
- Architect reusable agent systems, including multi-step workflows, LLM DAGs, conversational agents, and orchestration patterns.
- Develop and scale retrieval-augmented generation (RAG) systems, including vector database management and retrieval optimization.
- Integrate observability and monitoring tools for LLM-based systems to improve debugging, reliability, and performance tracking.
- Make informed technical decisions across LLM providers, frameworks, and tooling based on trade-offs in cost, latency, and reliability.
- Lead end-to-end delivery of AI engineering projects, from scoping through execution and deployment.
- Collaborate with engineering leadership to define AI system strategy and evaluation methodologies.
- Elevate engineering standards through code reviews, documentation, mentoring, and technical leadership.
- 5+ years of software engineering experience, with significant exposure to production AI/ML systems.
- Strong hands-on experience with LLM-based systems, including prompt engineering, RAG pipelines, and agent orchestration.
- Proven experience building and operating agentic AI systems in production environments.
- Deep understanding of AI evaluation frameworks and metrics, including the ability to distinguish meaningful signals from vanity metrics.
- Strong Python engineering skills with a focus on clean, testable, and production-grade code.
- Experience with tools such as LangGraph or similar agent orchestration frameworks.
- Familiarity with LLM observability tools (e.g., LangSmith or equivalents).
- Experience designing and working with vector databases and retrieval systems (e.g., Pinecone or similar).
- Strong understanding of cloud infrastructure (AWS, GCP, or equivalent), including serverless and event-driven architectures.
- Solid communication skills with the ability to discuss complex AI concepts across technical and non-technical audiences.
- Nice to have: experience with RLHF, LoRA, MLOps tooling, traditional ML, or open-source AI contributions.
- Competitive compensation with an estimated annual salary range of CAD $160,000–$180,000.
- Equity participation through stock options, subject to company plan terms.
- Comprehensive health benefits including medical, dental, life, disability, and AD&D coverage.
- Retirement savings plan and financial planning support.
- Paid time off, including holidays, sick leave, and generous parental leave.
- Remote-friendly work environment with home office and wellness stipends.
- Learning and development budget to support continuous skill growth.
- Wellness and mental health support programs and applications.
- Inclusive and accessible workplace with accommodations available upon request.
Requirements:
Benefits:
Explore More
Date Posted
05/28/2026
Views
0
Similar Jobs
Staff Supply Chain Security Engineer, Docker Hardened Images - Jobgether
Views in the last 30 days - 0
View DetailsSenior Sales Operations & Compensation Manager - Jobgether
Views in the last 30 days - 0
View Details