Honeycomb Observability Day SF - Kesha Mykhailov, Fin.ai: Human-Centric Observability in AI Systems

Empathy is one of the superpowers of modern #engineering teams, especially when building tools that interact with humans. This talk by Kesha Mykhailov tells the story of Fin, Intercom's Customer Support #ai agent, and how they transformed their approach to Fin's #observability.

Chapters

00:00 - Intro

00:36 - No one wants to talk about observability

01:10 - How Fin works

03:18 - The Fin customer experience is painfully slow

05:09 - A diagram of how Fin works

07:28 - Why you need distributed tracing for complex systems

10:18 - Time to first token, end-to-end latency

11:32 - If you liked it, then you should've put an SLO on it

12:52 - Realtime feedback on how A/B tests impact SLOs

14:02 - Finance enters the chat - business-centric cost signals

17:31 - How they solved their cost issues with traces

18:18 - Switching LLM providers

22:42 - Rasmussen safety model

24:07 - The art of observability

25:17 - Empathy is your superpower in the age of AI