Honeycomb CEO Christine Yen's KubeCon Europe keynote focused on how observability helps with LLMs in software systems.

• LLMs are different from "black boxes" as they make reliable behaviors more complicated.
• Current systems rely on deterministic properties but incorporate LLMs for unpredictable human language.
• Product release practices have shifted to early access programs with unpredictable user interactions.
• Practices like continuous deployment, testing in production, high cardinality metadata, high dimensionality data, and service level objectives are important.
• Observability compares expected and actual production behavior to handle early access program chaos.
• Tests are insufficient for LLMs; evaluations are more flexible and incorporated into the development loop.
• Similar to other black boxes, LLMs need to capture details for understanding system behavior.
• Observability helps embrace unpredictability and enables rapid investigation of outlier users.
  Yen is optimistic about software engineering's future as developers adapt to system behavior unpredictability, and observability is crucial in the age of Generative AI.