AI Testing Frameworks: Ensuring AI Application Quality

When writing regular programs, you can write unit tests and integration tests. But the output of AI applications is natural language, and the same prompt may return different results each time. How do you test that?

💡 What You Will Learn

When writing regular programs, you can write unit tests and integration tests. But the output of AI applications is natural language, and the same prompt may return different results each time. How do

def test_ai_responds():
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": ""}]
    )
    assert response.choices[0].message.content is not None
    assert len(response.choices[0].message.content) > 0

def test_quality():
    prompt = "RAG"
    response = call_llm(prompt)

    eval_prompt = f"""
{prompt}
{response}
1-10
"""
    score = call_llm(eval_prompt)
    assert int(score) >= 7

[
  {"question": "RAG", "expected_keywords": ["Retrieve", "Generate", ""]},
  {"question": "Ollama", "expected_keywords": ["ollama.com", "curl", "docker"]}
]

|:----|:----|