模型可解释性

对于模型可解释性的资料较少，这里统一做个整理。
如果想系统性学习，推荐阅读Interpretable machine learning 这本书。

What

可解释性是人可以理解系统决策的一个度量(Interpretability is the degree to which a human can understand the cause of a decision.)

Why

随着大量科学家在机器学习领域的持续投入，如今各种模型五花八门，且毋庸置疑这些模型在生产环境都产生的一定的效果。

但计算机系统做出的这些预测或决策的原因并无法被人类感知，因而人类无法直观的感知这些预测的效果或原因。

模型都是对现实世界的模拟，但是都不是 100% 的还原，因此没有一个单一 metric(如 auc) 能描述模型在生成环境中的真实表现。

此外机器学习系统并不知道自己在预测什么内容。

这个问题在某些领域可能非常重要（例如临床医学，反欺诈）。

How(理论)

• 使用有自解释性的模型(WB)
• 基于模型不可知论的黑盒探测(BB)
• 深度学习/神经网络模型的可解释性

自解释性的模型

• 线性回归模型
• 逻辑回归
• 决策树

白盒解释，需要知道模型具体实现，以及训练数据，甚至可能需要改模型代码。

黑盒探测

• Partial Dependence Plot (PDP)
• Global Surrogate
• Local Surrogate Model-Agnostic Method(LIME)
• Anchors(If-Then-That)
• Shapley Values
• SHAP

PDP

A partial dependence plot can show whether the relationship between the target and a feature is linear, monotonic or more complex.

Advantages: intuitive
Disadvantages:The assumption of independence is the biggest issue with PD plots.

Global Surrogate

A global surrogate model is an interpretable model that is trained to approximate the predictions of a black box model. 

Advantages: flexible.Any interpretable models can be used. 
Disadvantages:You have to be aware that you draw conclusions about the model and not about the data, since the surrogate model never sees the real outcome.

Local Surrogate Model-Agnostic Method(LIME)

Local surrogate models are interpretable models that are used to explain individual predictions of black box machine learning models.

Advantages: They make human-friendly explanations.LIME is one of the few methods that works for tabular data, text and images.
Disadvantages: The correct definition of the neighborhood differs./instability.

Anchor

A rule anchors a prediction if changes in other feature values do not affect the prediction. 
Anchors utilizes reinforcement learning techniques in combination with a graph search algorithm to reduce the number of model calls (and hence the required runtime) to a minimum while still being able to recover from local optima. 

Advantages: Like Lime. 
    Anchors are subsettable(shown as example).
    Works for non-linear or complex in an instance’s neighborhood(reinforcement learning).
    Can be parallelized.
    
Disadvantages: highly configurable.
    many scenarios require discretization.
    many calls to the ML model.

Shapley Values

SHAP

explore timeline

PDP(2001) --> LIME(2016) --> Anchors(2018)
Shapley Values(2014) --> SHAP(2016)

深度学习/神经网络模型

深度学习/神经网络模型的解释可以使用黑盒探测来实现，除此之外还有一些专门针对神经网络的可解释性科研探索。

• Feature Visualization
• Network dissection
• Pixel Attribution (Saliency Maps)

When to use

• Your model makes significant impact.
• When the problem is not well studied, or explore in a very new area.

In Action(实战)

python: sklearn、keras、alibi

R: iml

seldon 是一个模型生命周期管理的系统，有点类似于我们的 PAS 加一部分 DAG 的功能，他们基于上述 LIME 和 Anchor 等实现了一套模型解释和异常检测的框架并开源了，我们可以借鉴、探索下。

模型 outlier, adversarial and drift

https://github.com/SeldonIO/a...

模型 inspection and interpretation

https://github.com/SeldonIO/a...

具体参考 explain.ipynb