In this episode of TMI, Chip and Tess dive deep into the frustrating world of flaky tests. These are the tests that pass sometimes and fail at other times, even when the code hasn't changed.

We discuss reference research from Trunk, who analyzed over 20 million CI jobs and found that flaky tests are a widespread problem, even among large tech companies like Meta, Google, and Uber.

We examine strategies for managing flaky tests, such as quarantining them. We also cover how to prevent flaky tests by ensuring isolation, using dynamic waits instead of hard-coded timeouts, and utilizing deterministic inputs.

Finally, we discuss the importance of the testing pyramid, controlled test environments, contract testing, robust error handling, retry mechanisms, and clear communication.

References:

• https://www.testrail.com/blog/flaky-tests/
• https://testsigma.com/guides/flaky-test/
• https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-1
• https://trunk.io/blog/what-we-learned-from-analyzing-20-2-million-ci-jobs-in-trunk-flaky-tests-part-2

Explore the three pillars of Agile estimating and planning: embracing change, prioritizing value, and effective communication—principles that resonate beyond software development.

• Learn why multitasking can hinder progress despite feeling productive.
• Understand the pitfalls of setting initial estimates in stone.
• Learn why providing estimation dates and team velocity as ranges -- rather than a definitive number -- helps manage expectations.
• Discover how properly prioritizing features can deliver 80% of benefits with 20% of the effort.
• We discuss using Story Points to estimate complexity, Ideal Days for estimating the time it will take to completer a task, and tips for splitting large user stories.
• Delve into planning poker for estimation and strategies for backlog prioritization.

Finally, we consider a sprint structure that includes dedicated time for technical debt.

More Resources:

• Agile Estimating and Planning

In this episode, we dive into exploratory testing, a dynamic technique where tests are designed and executed simultaneously, using insights from previous experiments to guide future ones. We explore various exploratory testing tours, including:

• The Guide Book Tour: Following the user manual step by step.
• The Money Tour: Prioritizing features most essential to users.
• The Saboteur Tour: Intentionally introducing problems like network errors and corrupted files.
• The Anti-Social Tour: Doing the opposite of expected user actions.

We also discuss testing in the small versus testing in the large, scenario testing, and designing tests using states and events. Finally, we share thoughts on the future of testing.

Other Resources:

• Exploratory Software Testing
• Explore it? Explore it! (YouTube Video)

In this episode of the TMI Podcast, we dive into the intriguing world of Testing in Production and how it allows you to observe your software in real-world conditions. Join us as we discuss key techniques to minimize risks while testing in production:

• Feature Flags: Deploy code to production servers without immediate exposure to users. Feature flags allow you to unveil new features only when you’re confident they will function seamlessly.
• Shadowing: Safely test new code by sending a copy of live traffic without impacting the customer experience.
• Staged Rollouts and Canarying: Gradually introduce new code to a small percentage of users, increasing exposure as confidence grows.
• A/B Testing: Compare different versions of features to determine the most effective approach.

We also discuss the innovative practice of chaos engineering, where intentionally causing disruptions builds system resilience and confidence.

Discover why a corporate culture that embraces risk is essential for successful Testing in Production. Tune in for insights on how to leverage these strategies to enhance your software’s performance and reliability in the real world.

More Resources:

https://www.youtube.com/watch?v=n2iE0wUt3bc https://medium.com/life-at-apollo-division/testing-in-production-339dc58dd8cd https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-hard-parts-3f06cefaf592

Are you ready to navigate the AI revolution and elevate your career? Dive into our latest episode, where AI and testing experts share the must-have skills to thrive in the age of artificial intelligence.

🔍 Key Highlights:

• Essential AI Literacy: Discover why understanding AI is vital for everyone in tech, no matter your role.
• Transforming Testing: Learn how AI is reshaping the testing landscape and what this means for you.
• Human Skills Matter: Explore the crucial role of critical thinking, context understanding, and ethical judgment in an AI-driven world.
• Prompt Engineering Insights: Uncover the skills that truly matter when working with AI.
• New Opportunities: Find out how AI is creating job prospects and the importance of continuous learning.
• Building AI Literacy: Get practical steps to enhance your AI knowledge and stay ahead, featuring resources like Deepseek and Testers.ai.
• AI Assurance and Ethics: Understand the growing importance of responsible AI use and the human element in tech.

Don’t get left behind! Learn to collaborate with AI and position yourself for success in the tech landscape of 2025 and beyond. Tune in for actionable insights and a reassuring perspective on the future of work in the age of AI.

Resources:

• https://www.youtube.com/watch?v=FHS6hlDko4E
• https://www.youtube.com/watch?v=oX625Lm1QoM

In this episode, we explore the four essential metrics that define high-performing teams:

1. Delivery Lead Time: The speed of transitioning code from a developer’s machine to production.
2. Deployment Frequency: The regularity of deployments to production.
3. Resilience: Mean time to restore service
4. Change Fail Rate: The percentage of releases that encounter issues in production.

We challenge the notion that developers shouldn’t be highly engaged in testing. We also delve into the critical role of fast feedback loops.

Other Resources:

• Accelerate: Building and Scaling High Performing Technology Organizations
• The Weasel Speaks Blog