9.6. PCI NTB 功能¶
- 作者:
Kishon Vijay Abraham I <kishon@ti.com>
PCI 不透明桥 (NTB) 允许两个主机系统通过将每个主机作为设备暴露给另一个主机来进行通信。NTB 通常支持在远程机器上生成中断、将内存范围暴露为 BARs 并执行 DMA 的能力。它们还支持暂存器（scratchpad），这是 NTB 内部的内存区域，可从两台机器访问。
PCI NTB 功能允许两个不同的系统（或主机）通过配置端点实例来相互通信，从而将一个系统中的事务路由到另一个系统。
在下图中，PCI NTB 功能将 SoC 配置为多个 PCI 端点 (EP) 实例，使得来自一个 EP 控制器的事务被路由到另一个 EP 控制器。一旦 PCI NTB 功能将 SoC 配置为多个 EP 实例，HOST1 和 HOST2 就可以使用 SoC 作为桥梁进行相互通信。
+-------------+ +-------------+
| | | |
| HOST1 | | HOST2 |
| | | |
+------^------+ +------^------+
| |
| |
+---------|-------------------------------------------------|---------+
| +------v------+ +------v------+ |
| | | | | |
| | EP | | EP | |
| | CONTROLLER1 | | CONTROLLER2 | |
| | <-----------------------------------> | |
| | | | | |
| | | | | |
| | | SoC With Multiple EP Instances | | |
| | | (Configured using NTB Function) | | |
| +-------------+ +-------------+ |
+---------------------------------------------------------------------+
9.6.1. 用于实现 NTB 的构造¶
配置区域
自用暂存器
对等暂存器
门铃 (DB) 寄存器
内存窗口 (MW)
9.6.1.1. 配置区域:¶
配置区域是使用 NTB 端点功能驱动程序实现的 NTB 特有的构造。主机侧和端点侧的 NTB 功能驱动程序将使用此区域相互交换信息。配置区域具有用于配置端点控制器的控制/状态寄存器。主机可以写入此区域以配置出站地址转换单元 (ATU) 并指示链路状态。端点可以在此区域中指示主机发出的命令的状态。端点还可以使用此区域向主机指示暂存器偏移和内存窗口数量。
配置区域的格式如下所示。此处所有字段均为 32 位。
+------------------------+
| COMMAND |
+------------------------+
| ARGUMENT |
+------------------------+
| STATUS |
+------------------------+
| TOPOLOGY |
+------------------------+
| ADDRESS (LOWER 32) |
+------------------------+
| ADDRESS (UPPER 32) |
+------------------------+
| SIZE |
+------------------------+
| NO OF MEMORY WINDOW |
+------------------------+
| MEMORY WINDOW1 OFFSET |
+------------------------+
| SPAD OFFSET |
+------------------------+
| SPAD COUNT |
+------------------------+
| DB ENTRY SIZE |
+------------------------+
| DB DATA |
+------------------------+
| : |
+------------------------+
| : |
+------------------------+
| DB DATA |
+------------------------+
COMMAND:
NTB function supports three commands:
CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
invoking this command, the host should allocate and initialize
MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
Endpoint). The endpoint on receiving this command will configure
the outbound ATU such that transactions to Doorbell BAR will be routed
to the MSI/MSI-X address programmed by the host. The ARGUMENT
register should be populated with number of DBs to configure (in the
lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
host invokes this command after allocating a buffer that can be
accessed by remote host. The allocated address should be programmed
in the ADDRESS register (64 bit), the size should be programmed in
the SIZE register and the memory window index should be programmed
in the ARGUMENT register. The endpoint on receiving this command
will configure the outbound ATU such that transactions to MW BAR
are routed to the address provided by the host.
CMD_LINK_UP (0x3): Command to indicate an NTB application is
bound to the EP device on the host side. Once the endpoint
receives this command from both the hosts, the endpoint will
raise a LINK_UP event to both the hosts to indicate the host
NTB applications can start communicating with each other.
ARGUMENT:
The value of this register is based on the commands issued in
command register. See COMMAND section for more information.
TOPOLOGY:
Set to NTB_TOPO_B2B_USD for Primary interface
Set to NTB_TOPO_B2B_DSD for Secondary interface
ADDRESS/SIZE:
Address and Size to be used while configuring the memory window.
See "CMD_CONFIGURE_MW" for more info.
MEMORY WINDOW1 OFFSET:
Memory Window 1 and Doorbell registers are packed together in the
same BAR. The initial portion of the region will have doorbell
registers and the latter portion of the region is for memory window 1.
This register will specify the offset of the memory window 1.
NO OF MEMORY WINDOW:
Specifies the number of memory windows supported by the NTB device.
SPAD OFFSET:
Self scratchpad region and config region are packed together in the
same BAR. The initial portion of the region will have config region
and the latter portion of the region is for self scratchpad. This
register will specify the offset of the self scratchpad registers.
SPAD COUNT:
Specifies the number of scratchpad registers supported by the NTB
device.
DB ENTRY SIZE:
Used to determine the offset within the DB BAR that should be written
in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
ring doorbell (MSI-X support will be added later). MSI uses same
address for all the interrupts and MSI-X can provide different
addresses for different interrupts. The MSI/MSI-X address is provided
by the host and the address it gives is based on the MSI/MSI-X
implementation supported by the host. For instance, ARM platform
using GIC ITS will have the same MSI-X address for all the interrupts.
In order to support all the combinations and use the same mechanism
for both MSI and MSI-X, EPF NTB allocates a separate region in the
Outbound Address Space for each of the interrupts. This region will
be mapped to the MSI/MSI-X address provided by the host. If a host
provides the same address for all the interrupts, all the regions
will be translated to the same address. If a host provides different
addresses, the regions will be translated to different addresses. This
will ensure there is no difference while raising the doorbell.
DB DATA:
EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
This holds the MSI/MSI-X data that has to be written to MSI address
for raising doorbell interrupt. This will be populated by EPF NTB
while invoking CMD_CONFIGURE_DOORBELL.
9.6.1.2. 暂存器:¶
每个主机都在 NTB 端点控制器的内存中分配有自己的寄存器空间。它们可以从桥的两侧进行读写。它们被构建在 NTB 上的应用程序使用，可用于在设备的双方之间传递控制和状态信息。
- 暂存器有 2 部分
自用暂存器：主机的自身寄存器空间
对等暂存器：远程主机的寄存器空间。
9.6.1.3. 门铃寄存器:¶
门铃寄存器用于主机之间相互中断。
9.6.1.4. 内存窗口:¶
两个主机之间实际的数据传输将通过内存窗口进行。
9.6.2. 建模构造:¶
为了实现 NTB 功能，需要建模 5 个或更多不同的区域（配置、自用暂存器、对等暂存器、门铃、一个或多个内存窗口）。至少需要一个内存窗口，并允许有多个。所有这些区域都应映射到 BARs，以便主机访问这些区域。
如果为每个区域分配一个 32 位 BAR，方案将如下所示
BAR 编号 |
使用的构造 |
|---|---|
BAR0 |
配置区域 |
BAR1 |
自用暂存器 |
BAR2 |
对等暂存器 |
BAR3 |
门铃 |
BAR4 |
内存窗口 1 |
BAR5 |
内存窗口 2 |
但是，如果我们为每个区域分配一个单独的 BAR，在一个仅支持 64 位 BAR 的平台上，将没有足够的 BAR 来容纳所有区域。
为了得到大多数平台的支持，这些区域应以一种既能提供 NTB 功能又能确保主机不会访问不应访问的任何区域的方式进行打包并映射到 BARs。
EPF NTB 功能中采用以下方案
BAR 编号 |
使用的构造 |
|---|---|
BAR0 |
配置区域 + 自用暂存器 |
BAR1 |
对等暂存器 |
BAR2 |
门铃 + 内存窗口 1 |
BAR3 |
内存窗口 2 |
BAR4 |
内存窗口 3 |
BAR5 |
内存窗口 4 |
通过此方案，对于基本的 NTB 功能，3 个 BAR 应该足够了。
9.6.2.1. 配置/暂存器区域建模:¶
+-----------------+------->+------------------+ +-----------------+
| BAR0 | | CONFIG REGION | | BAR0 |
+-----------------+----+ +------------------+<-------+-----------------+
| BAR1 | | |SCRATCHPAD REGION | | BAR1 |
+-----------------+ +-->+------------------+<-------+-----------------+
| BAR2 | Local Memory | BAR2 |
+-----------------+ +-----------------+
| BAR3 | | BAR3 |
+-----------------+ +-----------------+
| BAR4 | | BAR4 |
+-----------------+ +-----------------+
| BAR5 | | BAR5 |
+-----------------+ +-----------------+
EP CONTROLLER 1 EP CONTROLLER 2
上图显示了为 HOST1（连接到 EP 控制器 1）在本地内存中分配的配置区域 + 暂存器区域。HOST1 可以使用 EP 控制器 1 的 BAR0 访问配置区域和暂存器区域（自用暂存器）。对等主机（连接到 EP 控制器 2 的 HOST2）也可以使用 EP 控制器 2 的 BAR1 访问此暂存器区域（对等暂存器）。此图显示了为 HOST1 分配配置区域和暂存器区域的情况，但同样适用于 HOST2。
9.6.2.2. 门铃/内存窗口 1 建模:¶
+-----------------+ +----->+----------------+-----------+-----------------+
| BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
+-----------------+ | +----------------+ +-----------------+
| BAR1 | | | Doorbell 2 +---------+ | |
+-----------------+----+ +----------------+ | | |
| BAR2 | | Doorbell 3 +-------+ | +-----------------+
+-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
| BAR3 | | | Doorbell 4 +-----+ | +-----------------+
+-----------------+ | |----------------+ | | | |
| BAR4 | | | | | | +-----------------+
+-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
| BAR5 | | | | | | +-----------------+
+-----------------+ +----->-----------------+ | | | |
EP CONTROLLER 1 | | | | +-----------------+
| | | +---->+ MSI-X ADDRESS 4 |
+----------------+ | +-----------------+
EP CONTROLLER 2 | | |
(OB SPACE) | | |
+-------> MW1 |
| |
| |
+-----------------+
| |
| |
| |
| |
| |
+-----------------+
PCI Address Space
(Managed by HOST2)
上图显示了门铃和内存窗口 1 如何映射，以便 HOST1 可以在 HOST2 上触发门铃中断，以及 HOST1 如何使用内存窗口 1 (MW1) 访问 HOST2 暴露的缓冲区。此处，门铃和内存窗口 1 区域分配在 EP 控制器 2 的出站 (OB) 地址空间中。为门铃和内存窗口 1 分配和配置 BARs 是在 NTB 端点功能驱动程序的初始化阶段完成的。从 EP 控制器 2 的 OB 空间到 PCI 地址空间的映射是在 HOST2 发送 CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL 时完成的。
9.6.2.3. 可选内存窗口建模:¶
这与 MW1 的建模方式相同，但每个额外的内存窗口都映射到单独的 BARs。