DeepSort之源码解读

文章有点长...

代码地址：https://github.com/ZQPei/deep...

traker是一个类，负责对多个track的进行操作，包括预测和更新。

self.tracker.predict()
self.tracker.update(detections)

tracker预测阶段是对每个track进行预测，包括

• 卡尔曼预测
• track年龄 age+1
• time_since_update+1，此变量用于记录track上次更新的时间

代码如下：

 def predict(self, kf):
        """Propagate the state distribution to the current time step using a
        Kalman filter prediction step.

        Parameters
        ----------
        kf : kalman_filter.KalmanFilter
            The Kalman filter.

        """
        self.mean, self.covariance = kf.predict(self.mean, self.covariance)
        self.age += 1
        self.time_since_update += 1

tracker更新是对多个track更新：

• track和det的匹配
• track更新
• 距离指标更新

代码如下：

 def update(self, detections):
        """Perform measurement update and track management.

        Parameters
        ----------
        detections : List[deep_sort.detection.Detection]
            A list of detections at the current time step.

        """
        # Run matching cascade.
        matches, unmatched_tracks, unmatched_detections = \
            self._match(detections)
        print("matches:",matches, "unmatched_tracks:",unmatched_tracks, "unmatched_detections:", unmatched_detections)

        # Update track set.
        for track_idx, detection_idx in matches:
            self.tracks[track_idx].update(
                self.kf, detections[detection_idx])
        for track_idx in unmatched_tracks:
            self.tracks[track_idx].mark_missed()
        for detection_idx in unmatched_detections:
            self._initiate_track(detections[detection_idx])
        self.tracks = [t for t in self.tracks if not t.is_deleted()]

        # Update distance metric.
        active_targets = [t.track_id for t in self.tracks if t.is_confirmed()]
        features, targets = [], []
        for track in self.tracks:
            if not track.is_confirmed():
                continue
            features += track.features
            print("1 features",track.track_id,np.array(features).shape)
            targets += [track.track_id for _ in track.features]
            print("1 targets_id",track.track_id,targets)
            track.features = []
        self.metric.partial_fit(
            np.asarray(features), np.asarray(targets), active_targets)

第一帧

检测结果如下：

det [array([307,  97, 105, 345]), array([546, 151,  72, 207]), array([215, 154,  59, 184]), array([400, 181,  45, 126])]

得到检测结果后进入track predict阶段，但是第一帧还没有track，所以没有predict结果。

#code1
for track in self.tracks:
            track.predict(self.kf)

接着进入track update阶段，首先对检测结果进行匹配

matches, unmatched_tracks, unmatched_detections = \
            self._match(detections)

在匹配中，首先要将track分成confirmed_track和unconfirmed_track，

  confirmed_tracks = [
            i for i, t in enumerate(self.tracks) if t.is_confirmed()]
        unconfirmed_tracks = [
            i for i, t in enumerate(self.tracks) if not t.is_confirmed()]

显然

confirmed_track: [] unconfirmed_track: []

对confirmed_track进行级连匹配

        matches_a, unmatched_tracks_a, unmatched_detections = \
            linear_assignment.matching_cascade(
                gated_metric, self.metric.matching_threshold, self.max_age,
                self.tracks, detections, confirmed_tracks)

显然，没有匹配的track，也没有没匹配的track，只有没匹配的检测。

matches_a [] unmatched_track_a [] unmatched_detections  [0, 1, 2, 3]

对所有检测创建新的track：

   for detection_idx in unmatched_detections:
            self._initiate_track(detections[detection_idx])

初始化代码如下：

def _initiate_track(self, detection):
        mean, covariance = self.kf.initiate(detection.to_xyah())
        self.tracks.append(Track(
            mean, covariance, self._next_id, self.n_init, self.max_age,
            detection.feature))
        self._next_id += 1

通过Track类初始化一个track， self._next_id += 1，因为创建一个track后，id也多一个了。

每个track初始化的属性如下：

self.mean = mean
self.covariance = covariance
self.track_id = track_id
self.hits = 1
self.age = 1
self.time_since_update = 0

self.state = TrackState.Tentative
self.features = []
if feature is not None:
    self.features.append(feature)

self._n_init = n_init
self._max_age = max_age

初始化的track，状态为Tentative，age=1，time_since_update = 0，features=[]。

默然3帧以内track的状态都是tentative。3帧以后便是conformed。30帧不更新则是deleted

第二帧

检测结果：

det [array([227, 152,  52, 189]), array([546, 153,  66, 203]), array([ 35,  52, 114, 466]), array([339, 130,  92, 278]), array([273, 134,  90, 268])]

因为第一帧得到了4个track，每个track进入predict阶段，进行卡尔曼预测，age和time_since_update都分别+1

  def predict(self, kf):
        """Propagate the state distribution to the current time step using a
        Kalman filter prediction step.

        Parameters
        ----------
        kf : kalman_filter.KalmanFilter
            The Kalman filter.

        """
        self.mean, self.covariance = kf.predict(self.mean, self.covariance)
        self.age += 1
        self.time_since_update += 1

此时，每个track的age和time_since_update分别为：

track update age 2 time_since_update 1
track update age 2 time_since_update 1
track update age 2 time_since_update 1
track update age 2 time_since_update 1

预测后进入track更新阶段

对预测的检测结果与之前得到的track进行匹配，首先将之前的track划分tracks为 confirmed_tracks 和unconfirmed_tracks，结果为：

confirmed_track [] unconfirmed_track [0, 1, 2, 3]

因confirmed_track为空，所以级联匹配结果为：

matches_a [] unmatched_track_a [] unmatched_detections  [0, 1, 2, 3,4]

接着，unconfirmed_track跟级联匹配结果的unmatched_track_a中time_since_update=1(上一帧得到更新)的track组成候选track。

   iou_track_candidates = unconfirmed_tracks + [
            k for k in unmatched_tracks_a if
            self.tracks[k].time_since_update == 1]

        unmatched_tracks_a = [
            k for k in unmatched_tracks_a if
            self.tracks[k].time_since_update != 1]

候选track跟unmatched_track_a结果为：

iou_track_candidates [0, 1, 2, 3],unmatched_track_a []

对候选track和没匹配的检测进行iou匹配

 matches_b, unmatched_tracks_b, unmatched_detections = \
            linear_assignment.min_cost_matching(
                iou_matching.iou_cost, self.max_iou_distance, self.tracks,
                detections, iou_track_candidates, unmatched_detections)

IOU匹配的结果为：

matches_b [(0, 0), (1, 1), (2, 2), (3, 3)] unmatches_track_b [] unmatched_detections [4]

最后将结果合并，级联匹配到的track跟iou匹配到track合并成最终的匹配结果，级联匹配中time_since_update!=1的track和iou没匹配到的track合并成最终的没匹配的track。可以看出，上一帧有更新的confirmed track会进行级联匹配和iou匹配，上一帧没更新的confirmed track会直接成为没匹配的track，从概率上说，上一帧有更新的track，当前帧会继续更新的概率会更大。

 matches = matches_a + matches_b
 unmatched_tracks = list(set(unmatched_tracks_a + unmatched_tracks_b))

最后结果为:

matches: [(0, 0), (1, 1), (2, 2), (3, 3)] unmatched_tracks: [] unmatched_detections: [4]

匹配完后，会有三种结果，分别是匹配到检测，未匹配到的track和未匹配的检测框。

接下来进入track数据更新阶段

对于匹配的结果，执行

 for track_idx, detection_idx in matches:
            self.tracks[track_idx].update(
                self.kf, detections[detection_idx])

每个track进行update

• 卡尔曼
• 检测边框特征，每个track都会存储一系列的特征，用作特征匹配
• hits
• time_since_update置0
• track状态，判断能够将状态设置为confirmed

   def update(self, kf, detection):
        """Perform Kalman filter measurement update step and update the feature
        cache.

        Parameters
        ----------
        kf : kalman_filter.KalmanFilter
            The Kalman filter.
        detection : Detection
            The associated detection.

        """
        self.mean, self.covariance = kf.update(
            self.mean, self.covariance, detection.to_xyah())
        self.features.append(detection.feature)

        self.hits += 1
        self.time_since_update = 0
        if self.state == TrackState.Tentative and self.hits >= self._n_init:
            self.state = TrackState.Confirmed

此时，所有track都能匹配到，他们的time_since_update都是0,。

对于未匹配到的track，对其状态进行标记，如果当前track状态为tentative，则该状态更新为deleted。如果太久没更新，time_since_update>max_age，该状态也将更新为deleted。

for track_idx in unmatched_tracks:
            self.tracks[track_idx].mark_missed()

 def mark_missed(self):
        """Mark this track as missed (no association at the current time step).
        """
        if self.state == TrackState.Tentative:
            self.state = TrackState.Deleted
        elif self.time_since_update > self._max_age:
            self.state = TrackState.Deleted

对于没有匹配到检测，创建新的track

   for detection_idx in unmatched_detections:
            self._initiate_track(detections[detection_idx])

然后检查所有track，将deleted状态的track删除。

 self.tracks = [t for t in self.tracks if not t.is_deleted()]

第三帧

检测结果为：

[array([307, 105, 108, 325]), array([547, 148,  70, 211]), array([216, 151,  59, 190]), array([402, 183,  43, 124]), array([ 35,  87,  70, 376])]

跟踪过程跟上一帧差不多，这里检测结果跟之前的track都能匹配上，track年龄和time_since_update为

track update age 3 time_since_update 1
track update age 3 time_since_update 1
track update age 3 time_since_update 1
track update age 3 time_since_update 1
track update age 2 time_since_update 1

匹配完之后，track set的更新会将部分track的状态更新为confirmed。

我们直接看第四帧。

第四帧

检测结果：

[array([318, 119, 105, 301]), array([545, 146,  71, 215]), array([216, 151,  59, 192]), array([ 30,  75,  82, 398]), array([403, 185,  41, 121])]

得到检测结果后进入预测阶段，track更新卡尔曼预测，age和time_since_update。

track update age 4 time_since_update 1
track update age 4 time_since_update 1
track update age 4 time_since_update 1
track update age 4 time_since_update 1
track update age 3 time_since_update 1

预测完成后进入track更新阶段

首先是检测的结果跟track匹配，在匹配中，要将track分成confirmed_track和unconfirmed_track，结果如下：

confirmed_t [0, 1, 2, 3] unconfirmed [4]

因为第4个det是第2帧才检出，所以状态还是unconfirmed。

接着对confirmed的track进行级联匹配

首先是对dets和confirmed_tracks创建索引

 if track_indices is None:
        track_indices = list(range(len(tracks)))
    if detection_indices is None:
        detection_indices = list(range(len(detections)))

结果为：

track_indices [0, 1, 2, 3] detection_indices [0, 1, 2, 3, 4]

当level=0时候，track_indices_l 索引中对应的time_since_update都是1，然后得到matches_l的匹配结果 ，当然level=1时候，track_indices_l 索引中对应的time_since_update都是2，然后再次得到匹配结果与之间结果进行合并，如此循环...，也就是先匹配最近有更新的track，由近到远...，保证了最近更新track的优先级。

 unmatched_detections = detection_indices
    matches = []
    for level in range(cascade_depth):
        if len(unmatched_detections) == 0:  # No detections left
            break

        track_indices_l = [
            k for k in track_indices
            if tracks[k].time_since_update == 1 + level
        ]
        if len(track_indices_l) == 0:  # Nothing to match at this level
            continue

        matches_l, _, unmatched_detections = \
            min_cost_matching(
                distance_metric, max_distance, tracks, detections,
                track_indices_l, unmatched_detections)
        matches += matches_l
    unmatched_tracks = list(set(track_indices) - set(k for k, _ in matches))

经过级联匹配后，得到的结果为：

matches_a [(0, 0), (1, 1), (2, 2), (3, 4)] unmatched_track_a [] unmatched_detections  [3]

剩下了一个没匹配的det。

unconfirmed_tracks和级联匹配中未匹配并且time_since_update =1的track组成了候选tracks。

iou_track_candidates [4]

候选tracks跟未匹配的det进行IOU匹配，结果如下：

matches_b [(4, 3)] unmatches_track_b [] unmatched_detections []

最终结果如下：

matches: [(0, 0), (1, 1), (2, 2), (3, 4), (4, 3)] unmatched_tracks: [] unmatched_detections: []

匹配结束后，将当前帧dets的feature更新到map(trackid->feature)中。

  active_targets = [t.track_id for t in self.tracks if t.is_confirmed()]
        features, targets = [], []
        for track in self.tracks:
            if not track.is_confirmed():
                continue
            features += track.features
            targets += [track.track_id for _ in track.features]
            track.features = []
        self.metric.partial_fit(
            np.asarray(features), np.asarray(targets), active_targets)

 def partial_fit(self, features, targets, active_targets):
        for feature, target in zip(features, targets):
            self.samples.setdefault(target, []).append(feature)
            if self.budget is not None:
                self.samples[target] = self.samples[target][-self.budget:]
        
        self.samples = {k: self.samples[k] for k in active_targets}

整个deepsort过程就这样子了，我们来看看更加细节的问题。

IOU匹配

如何得到代价矩阵？

初始化代价矩阵，矩阵(i,j)代表track i和det j的代价。然后计算卡尔曼滤波预测的bbx和det的IOU，代价=1-IOU。但是如果track已经有一帧以上(包含)没有更新，那么cost将会设置得很大，即为INFTY( 1e+5)。

def iou_cost(tracks, detections, track_indices=None,
             detection_indices=None):
    
   if track_indices is None:
        track_indices = np.arange(len(tracks))
    if detection_indices is None:
        detection_indices = np.arange(len(detections))

    cost_matrix = np.zeros((len(track_indices), len(detection_indices)))
    for row, track_idx in enumerate(track_indices):
        if tracks[track_idx].time_since_update > 1:
            cost_matrix[row, :] = linear_assignment.INFTY_COST
            continue

        bbox = tracks[track_idx].to_tlwh()
        candidates = np.asarray([detections[i].tlwh for i in detection_indices])
        cost_matrix[row, :] = 1. - iou(bbox, candidates)
    return cost_matrix

得到代价矩阵后，如果元素大于max_distance，该元素会设置为max_distance + 1e-5

cost_matrix[cost_matrix > max_distance] = max_distance + 1e-5

第二帧代价矩阵为：

[[0.04281178 1.         1.         0.96899767 1.        ]
 [1.         0.03566279 1.         1.         1.        ]
 [1.         1.         0.04389799 1.         1.        ]
 [0.95802783 1.         1.         0.08525083 1.        ]]
 
 #处理后
 [[0.04281178 0.70001    0.70001    0.70001    0.70001   ]
 [0.70001    0.03566279 0.70001    0.70001    0.70001   ]
 [0.70001    0.70001    0.04389799 0.70001    0.70001   ]
 [0.70001    0.70001    0.70001    0.08525083 0.70001   ]]

得到代价矩阵后，将其输入到匈牙利算法中

row_indices, col_indices = linear_assignment(cost_matrix)

当然也不是所有track和det都能得到匹配，iou匹配中把大于max_distacne的被认为是不匹配的。

 matches, unmatched_tracks, unmatched_detections = [], [], []
    for col, detection_idx in enumerate(detection_indices):
        if col not in col_indices:
            unmatched_detections.append(detection_idx)
    for row, track_idx in enumerate(track_indices):
        if row not in row_indices:
            unmatched_tracks.append(track_idx)
    for row, col in zip(row_indices, col_indices):
        track_idx = track_indices[row]
        detection_idx = detection_indices[col]
        if cost_matrix[row, col] > max_distance:
            unmatched_tracks.append(track_idx)
            unmatched_detections.append(detection_idx)
        else:
            matches.append((track_idx, detection_idx))

级联匹配

看看如何得到代价矩阵。一个track中保存了多个det的特征，所以该track跟当前帧某个det的特征会有多个余弦距离，取最小值作为该track与该det的最终余弦距离，然后再结合马氏矩阵进行处理。

def gated_metric(tracks, dets, track_indices, detection_indices):
            features = np.array([dets[i].feature for i in detection_indices])
            targets = np.array([tracks[i].track_id for i in track_indices])
            cost_matrix = self.metric.distance(features, targets) #计算代价矩阵
            cost_matrix = linear_assignment.gate_cost_matrix( #结合马氏矩阵进行处理
                self.kf, cost_matrix, tracks, dets, track_indices, #
                detection_indices)
            return cost_matrix

  def distance(self, features, targets):
        cost_matrix = np.zeros((len(targets), len(features)))
        for i, target in enumerate(targets):
            cost_matrix[i, :] = self._metric(self.samples[target], features)
        return cost_matrix

def _nn_cosine_distance(x, y):
    distances = _cosine_distance(x, y)
    return distances.min(axis=0) #取最小值

首先将det转换成xyah格式，

  measurements = np.asarray(
        [detections[i].to_xyah() for i in detection_indices])

接着计算track预测结果和检测结果的马氏距离，将马氏距离中大于gating_threshold( 9.4877 )的代价设置为gated_cost(100000.0)

for row, track_idx in enumerate(track_indices):
        track = tracks[track_idx]
        gating_distance = kf.gating_distance(
            track.mean, track.covariance, measurements, only_position)
        cost_matrix[row, gating_distance > gating_threshold] = gated_cost

最后将代价矩阵中大于max_distance的设置为max_distance(级接匹配中设为0.2) + 1e-5。

在第四帧中，余弦距离得到的代价矩阵为

[[0.02467382 0.29672492 0.14992237 0.20593166 0.25746107]
 [0.27289903 0.01389802 0.2490201  0.26275396 0.18523771]
 [0.1549592  0.25630915 0.00923228 0.10906434 0.27596951]
 [0.26783013 0.19509423 0.26934785 0.24842238 0.01052856]]

计算马氏距离，将马氏距离作用于余弦距离，将马氏大于gating_threshold的余弦代价设置为gated_cost(100000.0)。

然后得到的结果为

[[2.46738195e-02 1.00000000e+05 1.00000000e+05 1.00000000e+05
  1.00000000e+05]
 [1.00000000e+05 1.38980150e-02 1.00000000e+05 1.00000000e+05
  1.00000000e+05]
 [1.00000000e+05 1.00000000e+05 9.23228264e-03 1.00000000e+05
  1.00000000e+05]
 [1.00000000e+05 1.00000000e+05 1.00000000e+05 1.00000000e+05
  1.05285645e-02]]

代价矩阵中大于max_distance的设置为max_distance(级接匹配中设为0.2) + 1e-5，最终得到的代价矩阵为:

[[0.02467382 0.20001    0.20001    0.20001    0.20001   ]
 [0.20001    0.01389802 0.20001    0.20001    0.20001   ]
 [0.20001    0.20001    0.00923228 0.20001    0.20001   ]
 [0.20001    0.20001    0.20001    0.20001    0.01052856]]

然后将代价矩阵输入到匈牙利算法中求解。

deepsrot步骤如下

• track划分为uncomfirmed_track和comfirmed_track

• confirmed_track和det进行级联匹配

  • 1.计算track和检测结果的特征余弦距离cost matrix
  • 2.计算马氏距离，将马氏距离作用与cost matrix，若马氏距离中大于gating_threshold，cost matrix中相应的代价设置为gated_cost。
  • 3.将const matrix中大于max_distance的设置为max_distance
  • 4.匈牙利求解，删除匹配值较大的结果。
  • 根据track的time_since_update，循环1-4，并合并结果。

• unconfirmed_track和级联匹配中未能匹配并且time_since_update=1的track组成候选track，候选track和没匹配的det进行iou匹配

  • 对预测结果和检测结果计算iou代价矩阵
  • 匈牙利求解
• 合并级联匹配和iou匹配结果。

• 对于最终匹配到track进行以下操作

  • 卡尔曼更新
  • 存储边框特征
  • hits+1
  • time_since_update置0
  • track状态更新，判断能够将状态设置为confirmed

• 对于最终未能匹配到的track进行以下操作

  • 判断保留还是删除track，如果30帧没能更新，就删除。
• 对于最终未能匹配到的det创建新的track

整个流程如下图

ref:

Deep Sort 算法代码解读

[SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC](https:/