Reduce Test Flakiness: Best Practices

The "It Passed on My Machine" Problem

The test passed on your machine. You committed the code. CI failed. You re-ran it. It passed. You merged. Production deployment failed on the same test. You re-ran it again. It passed.

This is test flakiness, the silent killer of automation credibility.

I've watched engineering teams lose faith in their entire test suite because 5% of tests fail randomly. "Just re-run it" becomes the team's default response. When tests can't be trusted, they become noise instead of signal. Bugs slip through because nobody believes the failures anymore.

The worst part? Flaky tests aren't always obvious. They pass 95% of the time. Just often enough that you think they're reliable. Just rarely enough to wreck your deployment pipeline when you're on a deadline.

What Is Test Flakiness?

Test flakiness happens when a test produces inconsistent results without any changes to the code under test. The same test, the same code, different outcome.

A flaky test might pass 18 out of 20 times. Those two failures aren't random, they're symptoms of an underlying problem. The test is racing against something: a loading animation, a network request, a database query, an animation that hasn't finished.

According to our State of QA 2025 report, test maintenance, including fighting flakiness, consumes 40% of QA team time. That's hundreds of hours per year spent debugging tests instead of finding bugs.

A flaky test alternates between pass and fail across identical runs, while a stable test returns the same green result every single time.

The cost isn't just developer time. It's lost confidence. When developers stop trusting tests, they stop writing them.

Autonoma addresses flakiness at the root: its AI agents read your codebase, decide what needs testing, and generate tests that interact with your app the way real users do, on real browsers, with real waits, and no brittle selectors or hardcoded timing.

Root Causes: Where Flakiness Comes From

Flaky tests don't appear randomly. They emerge from predictable patterns.

Timing Issues and Race Conditions

Timing problems are the most common culprit. Your test clicks a button before the JavaScript has fully loaded. It checks for text before the API response has rendered. It interacts with an element before the animation completes.

Race conditions happen when tests assume things happen instantly. They don't. Networks have latency. Databases have query time. Browsers have render cycles.

Dynamic Content and Loading States

Modern web applications don't load all at once. They show spinners, skeleton screens, loading states, and progressive rendering. Your test needs to know when the actual content has loaded, not when the loading spinner appeared.

External Dependencies

Tests that call real APIs, connect to databases, or depend on third-party services inherit those systems' instability. The API times out occasionally. The database is under load. The third-party service rate limits you. Your test fails.

Poor Selectors

CSS selectors break. You target div > button:nth-child(3) and it works perfectly, until someone adds a button. Or changes the layout. Or uses a different CSS framework. The selector stops matching and the test fails.

These causes aren't independent. A test might use a brittle selector that works fine when the page loads quickly, but fails when there's network lag because it targets the wrong element during a loading state.

Start from the symptom your test shows and the decision tree points you to the most likely of the four root causes.

Flakiness Reduction Techniques: A Debugging Story

Let me show you how to fix flaky tests by walking through a real debugging session.

The Problem: A Login Test That Fails Randomly

I had a test that logged in, navigated to the dashboard, and verified the user's name appeared. It passed most of the time. Every fifth run, it failed with "Element not found."

Bad approach (what I tried first):

# BAD: Fixed sleep - The lazy solution
def test_login():
    driver.get("https://app.example.com/login")
    driver.find_element(By.ID, "email").send_keys("user@example.com")
    driver.find_element(By.ID, "password").send_keys("password123")
    driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click()
 
    time.sleep(5)  # "Just wait 5 seconds, that should be enough"
 
    username = driver.find_element(By.CSS_SELECTOR, "div.header > span:nth-child(2)").text
    assert username == "John Doe"

This "worked", for a while. Then it started failing again. Sometimes the page loaded in 3 seconds. Sometimes 7 seconds. The fixed sleep was a guess, not a solution.

Good approach (what actually fixed it):

# GOOD: Dynamic wait + stable selector
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def test_login():
    driver.get("https://app.example.com/login")
    driver.find_element(By.ID, "email").send_keys("user@example.com")
    driver.find_element(By.ID, "password").send_keys("password123")
    driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click()

    # Wait for the actual element we need, with a timeout
    wait = WebDriverWait(driver, 10)
    username_element = wait.until(
        EC.presence_of_element_located((By.ATTR, "data-testid", "user-name"))
    )

    assert username_element.text == "John Doe"

This version waits intelligently. It polls for the element every 500ms, up to 10 seconds. The moment the element appears, it proceeds. If the page loads in 2 seconds, the test waits 2 seconds. If it takes 8 seconds, the test waits 8 seconds.

The Selector Problem

The selector-stability lesson applies regardless of language or framework, so the examples below shift to Playwright and TypeScript to show the same principle in a second stack.

The CSS selector div.header > span:nth-child(2) was brittle. When the design team added an icon to the header, the span became the third child. The test broke.

The fix: Use data attributes specifically for testing.

Bad selector (brittle):

// BAD: Position-based selector that breaks with layout changes
const username = await page.locator('div.header > span:nth-child(2)').textContent();

Good selector (stable):

// GOOD: Test ID that survives design changes
const username = await page.locator('[data-testid="user-name"]').textContent();

Position-based CSS selectors break the moment the layout shifts, while a dedicated test ID keeps targeting the same element.

The test ID doesn't care about position, styling, or class names. It targets the specific element you need.

Network and API Waits

The test was also racing against an API call. The page rendered the header, but the API call to fetch the user's name hadn't completed. The element existed, but was empty.

Playwright solution (network idle):

// Wait for network to be idle before proceeding
await page.goto('https://app.example.com/dashboard', {
  waitUntil: 'networkidle'
});

// Or wait for a specific API call to complete
await page.waitForResponse(response =>
  response.url().includes('/api/user') && response.status() === 200
);

const username = await page.locator('[data-testid="user-name"]').textContent();
expect(username).toBe('John Doe');

A fixed sleep wastes time or fails early, while a dynamic wait resolves the instant the element is actually ready.

Best Practices for Stable Tests

1. Test Isolation

Each test should be completely independent. It should create its own test data, run, and clean up after itself.

Bad: Tests depend on each other:

# BAD: Test 2 depends on Test 1 creating the user
def test_1_create_user():
    create_user("test@example.com")
 
def test_2_login_user():
    login("test@example.com")  # Fails if test_1 didn't run first

Good: Each test is self-contained:

# GOOD: Each test sets up and tears down its own data
def test_create_user():
    user = create_user("create@example.com")
    assert user.email == "create@example.com"
    cleanup_user(user)

def test_login_user():
    user = create_user("login@example.com")  # Create fresh data
    login_result = login(user.email)
    assert login_result.success == True
    cleanup_user(user)

Use setup and teardown hooks:

import pytest

@pytest.fixture
def test_user():
    # Setup: Create user before test
    user = create_user(f"test_{uuid.uuid4()}@example.com")
    yield user
    # Teardown: Delete user after test
    cleanup_user(user)

def test_login(test_user):
    result = login(test_user.email)
    assert result.success == True

2. Idempotent Test Design

Tests should produce the same result no matter how many times you run them. They shouldn't fail because previous runs left data behind.

Use unique identifiers for test data:

// Generate unique test data each run
const testEmail = `test_${Date.now()}_${Math.random()}@example.com`;
const testProduct = `Test Product ${uuid()}`;

3. Retry Mechanisms

Some flakiness is unavoidable, network blips, infrastructure hiccups, cosmic rays. Retry mechanisms catch these edge cases without hiding real bugs.

# Retry flaky operations, not entire tests
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=10))
def fetch_user_data(user_id):
    response = requests.get(f"https://api.example.com/users/{user_id}")
    response.raise_for_status()
    return response.json()

Important: Retry the flaky operation (like an API call), not the entire test. If you retry the whole test, you might be hiding bugs instead of handling instability.

4. Explicit Waits Over Implicit Waits

Configure explicit waits for specific conditions, not global timeouts.

// BAD: Implicit wait applies to everything
await page.setDefaultTimeout(30000);
 
// GOOD: Explicit wait for specific condition
await page.waitForSelector('[data-testid="dashboard-loaded"]', {
  state: 'visible',
  timeout: 10000
});

5. Component Reusability

Extract common flows into reusable functions. When you fix flakiness in the function, every test that uses it benefits.

// Reusable login function with proper waits
async function login(page: Page, email: string, password: string) {
  await page.goto('/login');
  await page.fill('[data-testid="email"]', email);
  await page.fill('[data-testid="password"]', password);
  await page.click('[data-testid="login-button"]');

  // Wait for login to complete
  await page.waitForURL('/dashboard');
  await page.waitForSelector('[data-testid="user-menu"]', { state: 'visible' });
}

// Use it in every test that needs authentication
test('view profile', async ({ page }) => {
  await login(page, 'user@example.com', 'password123');
  await page.click('[data-testid="user-menu"]');
  await page.click('[data-testid="profile-link"]');
  // ... rest of test
});

When you discover a flakiness issue in login, you fix it once in the function, not in 47 different tests.

If you've already built out a Page Object Model per Chapter 5, this is the same principle applied to flows instead of pages: fix the flakiness once, in one place, and every test that depends on it benefits.

The Path Forward

Fighting flakiness is ongoing work. Every new feature, every UI change, every third-party integration introduces new opportunities for flakiness.

The techniques in this chapter, dynamic waits, stable selectors, test isolation, retry mechanisms, reduce flakiness from a constant problem to an occasional annoyance.

For hands-on setup, Chapter 7 walks through complete Python and JavaScript test automation projects where you can apply these practices directly.

But there's another approach emerging. Self-healing test automation uses AI to automatically detect and fix flakiness patterns. Instead of retrying or quarantining, self-healing tests adapt when the application changes, fixing broken selectors and reordered flows without human intervention.

We'll explore this in Chapter 8: How AI is changing test automation, including how autonomous testing handles flakiness automatically.

For now, apply these practices to your existing tests. Replace fixed sleeps with dynamic waits. Add test IDs to your application. Make your tests independent. Your future self (and your team) will thank you.

Frequently Asked Questions

Course Navigation

Chapter 6 of 8: Reduce Test Flakiness: Best Practices ✓

Next Chapter →

Chapter 7: Test Automation with Python and JavaScript

Get hands-on with complete setup guides for Python and TypeScript automation. Includes project structures, CI/CD configurations, and working examples you can use immediately.

← Previous Chapter

Chapter 5: Page Object Model & Test Architecture - Build maintainable test architecture