In a general consumer application, a recommendation system can be a black box. Nobody needs to know why the algorithm suggested that playlist. But in healthcare, legal practice, or financial services, the same opacity is a regulatory and ethical problem. If your model flags a patient for sepsis risk, the clinician needs to understand why — not just to comply with audit requirements, but because clinical judgment depends on it.
SHAP (SHapley Additive exPlanations) is the most versatile technique we reach for first. It assigns each input feature a contribution value for a specific prediction, grounded in game theory. For tabular models, SHAP produces clear, auditable explanations: 'The elevated lactate (contribution: +0.42) and low MAP (contribution: +0.31) drove this risk score.' This is interpretable to both engineers and clinicians.
LIME (Local Interpretable Model-agnostic Explanations) takes a different approach — it approximates the model locally with a simpler interpretable model around a specific prediction. LIME is useful when you need contrastive explanations ('what would need to change for this outcome to differ?') but produces less stable results than SHAP on complex models.
Attention maps and gradient-based methods (Grad-CAM, integrated gradients) are the tools of choice for image and text models. For a computer vision defect classifier, overlaying a Grad-CAM heatmap on the input image shows exactly which pixels drove the classification — essential for any human review step in a high-stakes pipeline.
