S1 E2 - Annie Vella - Proving What AI Builds cover art

S1 E2 - Annie Vella - Proving What AI Builds

S1 E2 - Annie Vella - Proving What AI Builds

Listen for free

View show details
Ray and John have a conversation with Annie Vella in this episode about why formal verification, a discipline she first encountered more than twenty years ago, is suddenly becoming increasingly relevant in the age of AI-assisted software development.With a background spanning mathematics, computer science, software engineering, DevOps, and large-scale banking systems, Annie Vella first studied formal methods at university in the early 2000s. At the time, proving software correct mathematically was intellectually rewarding but difficult, expensive, and rarely practical in everyday software development. Today, better tooling and the rapid rise of AI-generated code are changing that equation.We explore a growing problem in software engineering: AI can generate code dramatically faster, but that simply moves the bottleneck downstream. If humans remain responsible for reviewing every generated line, the promised productivity gains become difficult to realize. Formal verification offers another possibility. Rather than manually inspecting everything an AI produces, can we mathematically verify the properties that actually matter?That leads us into the importance of specifications. Annie discusses the renewed interest in spec-driven development, the difference between lengthy natural-language requirements and genuinely formal specifications, and how tools such as Dafny, Verus, Lean, SMT solvers, and emerging AI-assisted approaches could make rigorous verification more accessible to everyday engineers.However, verifying AI-generated code is only part of the challenge. Agentic systems introduce an even deeper problem: agents can dynamically make decisions, invoke tools, generate code, and take actions at runtime. Annie offers a deceptively simple example that an agent instructed to delete duplicate files could technically satisfy that requirement by deleting every file. Correctness therefore depends not only on verifying an outcome, but on precisely defining what the system is and is not allowed to do.We also examine what this could mean for highly regulated environments such as banking. Formal methods may provide a way to define critical invariants, constrain dangerous agent behavior, and even help modernize legacy systems by formally capturing essential properties before replacing decades-old implementations.Ultimately, our conversation comes back to something deeper than tooling. Building trustworthy AI-enabled systems requires engineers to become better at defining what they actually mean by “correct.” Formal verification forces that clarity. As AI makes generating software increasingly cheap, the ability to specify, verify, and reason about what that software should do may become one of the most valuable engineering skills of all.Show Notes:Birgitta Böckeler and Laura Tacho on Spec Driven Developmenthttps://www.thoughtworks.com/insights/podcasts/technology-podcasts/what-is-spec-driven-developmentDeveloping provably correct Rust code with Verushttps://www.amazon.science/blog/developing-provably-correct-rust-code-with-verusDafnyhttps://dafny.org/LemmaScripthttps://lemmascript.org/Guardians of the Agents: Formal verification of AI workflows by Erik Meijerhttps://dl.acm.org/doi/10.1145/3777544Fermat's Last Theorem, Xena project commentary on Anthropic's proofhttps://xenaproject.wordpress.com/2026/09/04/flt-anthropic-has-beaten-me-to-it/Easy Approach to Requirements Syntax (EARS)https://en.wikipedia.org/wiki/Easy_Approach_to_Requirements_Syntaxhttps://kiro.dev/blog/deep-spec-analysis/https://www.amazon.science/blog/developing-provably-correct-rust-code-with-verushttps://cdn.amazon.science/bb/40/22ac44f84f6d8eb625ac9666a00f/formally-verified-cloud-scale-authorization.pdfhttps://midspiral.comhttps://dl.acm.org/doi/pdf/10.1145/3815784https://dl.acm.org/doi/10.1145/3777544https://kiro.dev/docs/specs/feature-specs/https://dl.acm.org/doi/abs/10.1145/3815784
adbl_web_anon_alc_button_suppression_t1
No reviews yet