H3C BRAS 设备 IPv6 NDP 协议故障排除

园区网大扁平架构下，通过 BRAS 进行 IPoE 认证是目前园区网的主流架构，在这种架构下 IPv4 网络相关技术都比较成熟，但 IPv6 相关技术细节上目前还存在很多问题。

本文所涉及故障为 H3C SR8808-X 与 H3C S10508X-V 单臂互联的大扁平网络架构，用户 VLAN 采用 QinQ 方案，在 BRAS 设备上终结，通过 IPoE 进行 Web Portal 认证。此方案下 IPv4 网络一切正常，但 IPv6 接入终端设备虽能正确配置 IPv6 地址，却无法进行任何 IPv6 通信。此时如利用 ping 命令向 BRAS 设备接口的 global 地址发送报文后，接入终端设备就能够正常通信。以下是对该故障的分析和解决方案。

故障情况下 BRAS 设备接口配置：

interface Route-Aggregation1.4000
 description Uplink to CS105-01_Bridge-Aggregation 100
 ip address 10.151.255.254 255.255.0.0
 local-proxy-arp enable ip-range 10.151.0.0 to 10.151.255.255
 ip subscriber initiator arp enable
 user-vlan dot1q vid 4001 to 4021 second-dot1q 4000
 vlan-termination user-mode
 dhcp relay source-address 201.116.59.2
 dhcp select relay
 dhcp session-mismatch action fast-renew
 dhcp relay server-address 201.116.49.91
 dhcp flood-protection enable
 ipv6 address 2001:DB9:9001:8011::1/64
 undo ipv6 nd ra halt
 ipv6 nd ra dns server 2001:DB9:9001::110 sequence 10
 ipv6 nd ra dns server 2001:DB9:9001:1::10 sequence 20
 ip subscriber l2-connected enable ipv4
 ip subscriber authentication-method web mac-auth
 ip subscriber roaming enable
 ip subscriber pre-auth domain domain1
 ip subscriber web-auth domain domain2
 ip subscriber initiator unclassified-ip enable matching-user

从 BRAS 设备角度来看，业务接口上配置了 Dot1q 终结/QinQ 终结功能后是不能发送广播/组播报文的（RA 报文是否例外还需确认），所以 BRAS 设备无法发出组播的 NS 报文，也就无法主动更新接入终端设备相关的 neighbor cache 表项。

从接入终端设备角度来看，可以发送 NS 组播报文，并且可以被 BRAS 接受，但不会被再次广播出去。在这种情况下，终端设备实际上是可以主动的发起 NDP 协议相关的请求（NS RS）并正常更新 neighbor cache 表项，所有的 IPv6 会话也都是可以正常发起的，但无法收到任何返回的数据包。

关键问题是，接入终端设备的缺省路由是 BRAS 设备接口的 link-local 地址，接入终端设备并不知道 BRAS 设备接口的 global 地址。在正常情况下，接入终端设备发起的 NDP 协议相关请求都是与其 link-local 地址相关的，此时 BRAS 设备作为网关无法从 NDP 会话中获得接入终端设备 global 地址相关的任何信息，故而无法更新 BRAS 设备上接入终端设备 global 地址相关的 neighbor cache 表项，也就无法转发任何与其相关的下行 IPv6 报文了。此时可以看到 BRAS 设备的 neighbor cache 中有大量的 INCOMPLETE 表项。

接入终端设备情况：

$ ping6 www.163.com
PING6(56=40+8+8 bytes) 2001:db9:9001:8011:e47f:afb5:9f90:10d6 --> 240e:918:8007::1:fc
ping6: sendmsg: No route to host
ping6: wrote z163picipv6.v.bsgslb.cn 16 chars, ret=-1
^C
--- z163picipv6.v.bsgslb.cn ping6 statistics ---
1102 packets transmitted, 0 packets received, 100.0% packet loss

$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=400<CHANNEL_IO>
    ether 78:4f:43:66:a2:61
    inet6 fe80::c7b:a23a:608b:ff61%en0 prefixlen 64 secured scopeid 0x4
    inet6 2001:db9:9001:8011:183b:8734:928e:ba5f prefixlen 64 autoconf secured
    inet6 2001:db9:9001:8011:e47f:afb5:9f90:10d6 prefixlen 64 autoconf temporary
    inet 10.151.167.97 netmask 0xffff0000 broadcast 10.151.255.255
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

$ netstat -f inet6 -nr
Routing tables

Internet6:
Destination                             Gateway ...
default                                 fe80::9e54:c2ff:fe04:4002%en0...  
...
2001:db9:9001:8011::/64                 link#4 ...   

$ ndp -an
Neighbor                                Linklayer Address ...
fe80::9e54:c2ff:fe04:4002%en0           9c:54:c2:4:40:2 ...
...

BRAS 设备 neighbor cache 情况：

<BRAS>disp ipv6 neighbors all
...
IPv6 address              MAC address ... State ...
...
2001:DB9:9001:8011:E47F:AFB5:9F90:10D6 0000-0000-0000 ... INCMP ...
...
FE80::C7B:A23A:608B:FF61  784f-4366-a261 ... STALE ...
...

此时 NDP 相关会话（单播 NS 和 NA 报文）如下：

Frame 1: 86 bytes on wire (688 bits), 86 bytes captured (688 bits)
Ethernet II, Src: Apple_66:a2:61 (78:4f:43:66:a2:61), Dst: NewH3CTe_04:40:02 (9c:54:c2:04:40:02)
Internet Protocol Version 6, Src: fe80::c7b:a23a:608b:ff61, Dst: fe80::9e54:c2ff:fe04:4002
Internet Control Message Protocol v6
    Type: Neighbor Solicitation (135)
    Code: 0
    Checksum: 0xd0af [correct]
    [Checksum Status: Good]
    Reserved: 00000000
    Target Address: fe80::9e54:c2ff:fe04:4002
    ICMPv6 Option (Source link-layer address : 78:4f:43:66:a2:61)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: Apple_66:a2:61 (78:4f:43:66:a2:61)

Frame 2: 86 bytes on wire (688 bits), 86 bytes captured (688 bits)
Ethernet II, Src: NewH3CTe_04:40:02 (9c:54:c2:04:40:02), Dst: Apple_66:a2:61 (78:4f:43:66:a2:61)
Internet Protocol Version 6, Src: fe80::9e54:c2ff:fe04:4002, Dst: fe80::c7b:a23a:608b:ff61
Internet Control Message Protocol v6
    Type: Neighbor Advertisement (136)
    Code: 0
    Checksum: 0xae6a [correct]
    [Checksum Status: Good]
    Flags: 0xe0000000, Router, Solicited, Override
    Target Address: fe80::9e54:c2ff:fe04:4002
    ICMPv6 Option (Target link-layer address : 9c:54:c2:04:40:02)
        Type: Target link-layer address (2)
        Length: 1 (8 bytes)
        Link-layer address: NewH3CTe_04:40:02 (9c:54:c2:04:40:02)

关于单播 NS 报文的情况见 RFC 4861 section-7.3.1：

In some cases (e.g., UDP-based protocols and routers forwarding packets to hosts), such reachability information may not be readily available from upper-layer protocols. When no hints are available and a node is sending packets to a neighbor, the node actively probes the neighbor using unicast Neighbor Solicitation messages to verify that the forward path is still working.

针对以上情况，如果接入终端设备利用 ping 命令向网关 BRAS 设备接口的 global 地址发送 ICMP 包，情况就会发生变化。接入终端设备首先会以自己的 global 地址为源地址，针对 BRAS 设备接口的 global 地址发起组播 NS 请求，这个请求会被 BRAS 设备收到，这样 BRAS 设备就可以顺利更新接入终端设备 global 地址相关的 neighbor cache 表项了，此时接入终端设备的下行 IPv6 报文就可以顺利转发了。

接入终端设备情况：

$ ping6 2001:db9:9001:8011::1
PING6(56=40+8+8 bytes) 2001:db9:9001:8011:e47f:afb5:9f90:10d6 --> 2001:db9:9001:8011::1
16 bytes from 2001:db9:9001:8011::1, icmp_seq=0 hlim=64 time=5.728 ms
16 bytes from 2001:db9:9001:8011::1, icmp_seq=1 hlim=64 time=2.653 ms
16 bytes from 2001:db9:9001:8011::1, icmp_seq=2 hlim=64 time=2.938 ms
16 bytes from 2001:db9:9001:8011::1, icmp_seq=3 hlim=64 time=2.870 ms
^C
--- 2001:db9:9001:8011::1 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 2.653/3.547/5.728/1.263 ms

$ ping6 www.163.com
PING6(56=40+8+8 bytes) 2001:db9:9001:8011:e47f:afb5:9f90:10d6 --> 240e:918:8007::1:fe
16 bytes from 240e:918:8007::1:fe, icmp_seq=0 hlim=45 time=47.849 ms
16 bytes from 240e:918:8007::1:fe, icmp_seq=1 hlim=45 time=48.634 ms
16 bytes from 240e:918:8007::1:fe, icmp_seq=2 hlim=45 time=48.678 ms
16 bytes from 240e:918:8007::1:fe, icmp_seq=3 hlim=45 time=48.497 ms
^C
--- z163picipv6.v.bsgslb.cn ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 47.849/48.415/48.678/0.333 ms

$ ndp -an
Neighbor                                Linklayer Address ...
2001:db9:9001:8011::1                   9c:54:c2:4:40:2 ...
...
fe80::9e54:c2ff:fe04:4002%en0           9c:54:c2:4:40:2 ...
...

BRAS 设备 neighbor cache 情况：

<BRAS>disp ipv6 neighbors all
...
IPv6 address              MAC address ... State ...
...
2001:DB9:9001:8011:E47F:AFB5:9F90:10D6 784f-4366-a261 ... STALE ...

此时 NDP 相关会话（组播 NS 和 NA 报文）如下：

Frame 279: 86 bytes on wire (688 bits), 86 bytes captured (688 bits) on interface en0, id 0
Ethernet II, Src: Apple_66:a2:61 (78:4f:43:66:a2:61), Dst: IPv6mcast_ff:00:00:01 (33:33:ff:00:00:01)
Internet Protocol Version 6, Src: 2001:db9:9001:8011:e47f:afb5:9f90:10d6, Dst: ff02::1:ff00:1
Internet Control Message Protocol v6
    Type: Neighbor Solicitation (135)
    Code: 0
    Checksum: 0x5b71 [correct]
    [Checksum Status: Good]
    Reserved: 00000000
    Target Address: 2001:db9:9001:8011::1
    ICMPv6 Option (Source link-layer address : 78:4f:43:66:a2:61)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: Apple_66:a2:61 (78:4f:43:66:a2:61)

Frame 280: 86 bytes on wire (688 bits), 86 bytes captured (688 bits) on interface en0, id 0
Ethernet II, Src: NewH3CTe_04:40:02 (9c:54:c2:04:40:02), Dst: Apple_66:a2:61 (78:4f:43:66:a2:61)
Internet Protocol Version 6, Src: 2001:db9:9001:8011::1, Dst: 2001:db9:9001:8011:e47f:afb5:9f90:10d6
Internet Control Message Protocol v6
    Type: Neighbor Advertisement (136)
    Code: 0
    Checksum: 0xf974 [correct]
    [Checksum Status: Good]
    Flags: 0xe0000000, Router, Solicited, Override
    Target Address: 2001:db9:9001:8011::1
    ICMPv6 Option (Target link-layer address : 9c:54:c2:04:40:02)
        Type: Target link-layer address (2)
        Length: 1 (8 bytes)
        Link-layer address: NewH3CTe_04:40:02 (9c:54:c2:04:40:02)

H3C 文档：

vlan-termination broadcast enable 命令用来配置允许当前接口发送广播和组播报文，即允许当前接口遍历模糊终结的范围发送报文，具体为当前接口遍历模糊终结范围内的VLAN ID，给报文分别添加这些 VLAN ID 对应的 VLAN Tag 后发送。

接口上配置了可以终结多种 VLAN 报文的 Dot1q 终结/QinQ 终结功能后，不允许发送广播/组播报文。

针对以上情况，结合设备文档，只能在 BRAS 设备相关接口下增加如下配置：

interface Route-Aggregation1.4000
 ...
 vlan-termination broadcast enable
 ...

这样所有的广播/组播报文能够顺利发出来，NDP 相关协议才能够正常工作。

由于根据以上分析是 BRAS 设备的 NS 报文受到影响，类似命令 vlan-termination broadcast ra 并不能解决问题，经实测情况属实，只能放开所有广播报文。

此解决方案将全部广播包都放行，在极端情况下会加大设备负担，影响性能，后续希望厂商会有针对性的给出更好的解决方案。