9.6. PCI NTB 功能¶
- 作者:
Kishon Vijay Abraham I <kishon@ti.com>
PCI 非透明桥 (NTB) 允许两个主机系统通过将每个主机作为设备暴露给另一个主机来实现相互通信。NTB 通常支持在远程机器上生成中断、将内存范围暴露为 BAR 并执行 DMA 的能力。它们还支持便笺区，这是 NTB 内可从两台机器访问的内存区域。
PCI NTB 功能允许两个不同的系统（或主机）通过配置端点实例来实现相互通信，以便将一个系统的事务路由到另一个系统。
在下图中，PCI NTB 功能通过配置 SoC 中的多个 PCI 端点 (EP) 实例，使得来自一个 EP 控制器的事务路由到另一个 EP 控制器。一旦 PCI NTB 功能配置了具有多个 EP 实例的 SoC，HOST1 和 HOST2 就可以使用 SoC 作为桥梁进行相互通信。
+-------------+ +-------------+
| | | |
| HOST1 | | HOST2 |
| | | |
+------^------+ +------^------+
| |
| |
+---------|-------------------------------------------------|---------+
| +------v------+ +------v------+ |
| | | | | |
| | EP | | EP | |
| | CONTROLLER1 | | CONTROLLER2 | |
| | <-----------------------------------> | |
| | | | | |
| | | | | |
| | | SoC With Multiple EP Instances | | |
| | | (Configured using NTB Function) | | |
| +-------------+ +-------------+ |
+---------------------------------------------------------------------+
9.6.1. 用于实现 NTB 的构造¶
配置区域
自身便笺寄存器
对等便笺寄存器
门铃 (DB) 寄存器
内存窗口 (MW)
9.6.1.1. 配置区域：¶
配置区域是使用 NTB 端点功能驱动程序实现的 NTB 特有的构造。主机和端点侧的 NTB 功能驱动程序将使用此区域相互交换信息。配置区域具有用于配置端点控制器的控制/状态寄存器。主机可以写入此区域以配置出站地址转换单元 (ATU) 并指示链路状态。端点可以指示主机在此区域中发布的命令的状态。端点还可以使用此区域向主机指示便笺区偏移量和内存窗口的数量。
配置区域的格式如下所示。这里的所有字段都是 32 位。
+------------------------+
| COMMAND |
+------------------------+
| ARGUMENT |
+------------------------+
| STATUS |
+------------------------+
| TOPOLOGY |
+------------------------+
| ADDRESS (LOWER 32) |
+------------------------+
| ADDRESS (UPPER 32) |
+------------------------+
| SIZE |
+------------------------+
| NO OF MEMORY WINDOW |
+------------------------+
| MEMORY WINDOW1 OFFSET |
+------------------------+
| SPAD OFFSET |
+------------------------+
| SPAD COUNT |
+------------------------+
| DB ENTRY SIZE |
+------------------------+
| DB DATA |
+------------------------+
| : |
+------------------------+
| : |
+------------------------+
| DB DATA |
+------------------------+
COMMAND:
NTB function supports three commands:
CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
invoking this command, the host should allocate and initialize
MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
Endpoint). The endpoint on receiving this command will configure
the outbound ATU such that transactions to Doorbell BAR will be routed
to the MSI/MSI-X address programmed by the host. The ARGUMENT
register should be populated with number of DBs to configure (in the
lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
host invokes this command after allocating a buffer that can be
accessed by remote host. The allocated address should be programmed
in the ADDRESS register (64 bit), the size should be programmed in
the SIZE register and the memory window index should be programmed
in the ARGUMENT register. The endpoint on receiving this command
will configure the outbound ATU such that transactions to MW BAR
are routed to the address provided by the host.
CMD_LINK_UP (0x3): Command to indicate an NTB application is
bound to the EP device on the host side. Once the endpoint
receives this command from both the hosts, the endpoint will
raise a LINK_UP event to both the hosts to indicate the host
NTB applications can start communicating with each other.
ARGUMENT:
The value of this register is based on the commands issued in
command register. See COMMAND section for more information.
TOPOLOGY:
Set to NTB_TOPO_B2B_USD for Primary interface
Set to NTB_TOPO_B2B_DSD for Secondary interface
ADDRESS/SIZE:
Address and Size to be used while configuring the memory window.
See "CMD_CONFIGURE_MW" for more info.
MEMORY WINDOW1 OFFSET:
Memory Window 1 and Doorbell registers are packed together in the
same BAR. The initial portion of the region will have doorbell
registers and the latter portion of the region is for memory window 1.
This register will specify the offset of the memory window 1.
NO OF MEMORY WINDOW:
Specifies the number of memory windows supported by the NTB device.
SPAD OFFSET:
Self scratchpad region and config region are packed together in the
same BAR. The initial portion of the region will have config region
and the latter portion of the region is for self scratchpad. This
register will specify the offset of the self scratchpad registers.
SPAD COUNT:
Specifies the number of scratchpad registers supported by the NTB
device.
DB ENTRY SIZE:
Used to determine the offset within the DB BAR that should be written
in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
ring doorbell (MSI-X support will be added later). MSI uses same
address for all the interrupts and MSI-X can provide different
addresses for different interrupts. The MSI/MSI-X address is provided
by the host and the address it gives is based on the MSI/MSI-X
implementation supported by the host. For instance, ARM platform
using GIC ITS will have the same MSI-X address for all the interrupts.
In order to support all the combinations and use the same mechanism
for both MSI and MSI-X, EPF NTB allocates a separate region in the
Outbound Address Space for each of the interrupts. This region will
be mapped to the MSI/MSI-X address provided by the host. If a host
provides the same address for all the interrupts, all the regions
will be translated to the same address. If a host provides different
addresses, the regions will be translated to different addresses. This
will ensure there is no difference while raising the doorbell.
DB DATA:
EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
This holds the MSI/MSI-X data that has to be written to MSI address
for raising doorbell interrupt. This will be populated by EPF NTB
while invoking CMD_CONFIGURE_DOORBELL.
9.6.1.2. 便笺寄存器：¶
每个主机在 NTB 端点控制器的内存中都有自己分配的寄存器空间。它们可以从桥的两侧读取和写入。NTB 上构建的应用程序使用它们，可以用于在设备的双方之间传递控制和状态信息。
- 便笺寄存器有 2 个部分
自身便笺：主机自己的寄存器空间
对等便笺：远程主机的寄存器空间。
9.6.1.3. 门铃寄存器：¶
门铃寄存器由主机用于相互中断。
9.6.1.4. 内存窗口：¶
两个主机之间实际的数据传输将使用内存窗口进行。
9.6.2. 建模构造：¶
要实现 NTB 功能，需要建模 5 个或更多不同的区域（配置、自身便笺、对等便笺、门铃、一个或多个内存窗口）。至少需要一个内存窗口，但允许有多个。所有这些区域都应映射到 BAR，以便主机访问这些区域。
如果为每个区域分配一个 32 位 BAR，则方案将如下所示
BAR 编号 |
使用的构造 |
|---|---|
BAR0 |
配置区域 |
配置 |
自身便笺 |
BAR2 |
对等便笺 |
BAR3 |
门铃 |
BAR4 |
内存窗口 1 |
BAR5 |
内存窗口 2 |
但是，如果我们为每个区域分配一个单独的 BAR，则在仅支持 64 位 BAR 的平台上，将没有足够的 BAR 用于所有区域。
为了得到大多数平台的支持，应将这些区域打包并映射到 BAR，以提供 NTB 功能，并确保主机不会访问任何不应访问的区域。
以下方案用于 EPF NTB 功能
BAR 编号 |
使用的构造 |
|---|---|
BAR0 |
配置区域 + 自身便笺 |
配置 |
对等便笺 |
BAR2 |
门铃 + 内存窗口 1 |
BAR3 |
内存窗口 2 |
BAR4 |
内存窗口 3 |
BAR5 |
内存窗口 4 |
使用此方案，对于基本的 NTB 功能，3 个 BAR 应该就足够了。
9.6.2.1. 建模配置/便笺区域：¶
+-----------------+------->+------------------+ +-----------------+
| BAR0 | | CONFIG REGION | | BAR0 |
+-----------------+----+ +------------------+<-------+-----------------+
| BAR1 | | |SCRATCHPAD REGION | | BAR1 |
+-----------------+ +-->+------------------+<-------+-----------------+
| BAR2 | Local Memory | BAR2 |
+-----------------+ +-----------------+
| BAR3 | | BAR3 |
+-----------------+ +-----------------+
| BAR4 | | BAR4 |
+-----------------+ +-----------------+
| BAR5 | | BAR5 |
+-----------------+ +-----------------+
EP CONTROLLER 1 EP CONTROLLER 2
上图显示了分配在本地内存中的 HOST1（连接到 EP 控制器 1）的配置区域 + 便笺区域。HOST1 可以使用 EP 控制器 1 的 BAR0 访问配置区域和便笺区域（自身便笺）。对等主机（连接到 EP 控制器 2 的 HOST2）也可以使用 EP 控制器 2 的 BAR1 访问此便笺区域（对等便笺）。此图显示了为 HOST1 分配配置区域和便笺区域的情况，但同样适用于 HOST2。
9.6.2.2. 建模门铃/内存窗口 1：¶
+-----------------+ +----->+----------------+-----------+-----------------+
| BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
+-----------------+ | +----------------+ +-----------------+
| BAR1 | | | Doorbell 2 +---------+ | |
+-----------------+----+ +----------------+ | | |
| BAR2 | | Doorbell 3 +-------+ | +-----------------+
+-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
| BAR3 | | | Doorbell 4 +-----+ | +-----------------+
+-----------------+ | |----------------+ | | | |
| BAR4 | | | | | | +-----------------+
+-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
| BAR5 | | | | | | +-----------------+
+-----------------+ +----->-----------------+ | | | |
EP CONTROLLER 1 | | | | +-----------------+
| | | +---->+ MSI-X ADDRESS 4 |
+----------------+ | +-----------------+
EP CONTROLLER 2 | | |
(OB SPACE) | | |
+-------> MW1 |
| |
| |
+-----------------+
| |
| |
| |
| |
| |
+-----------------+
PCI Address Space
(Managed by HOST2)
上图显示了如何映射门铃和内存窗口 1，以便 HOST1 可以在 HOST2 上触发门铃中断，以及 HOST1 如何使用内存窗口 1 (MW1) 访问 HOST2 公开的缓冲区。这里的门铃和内存窗口 1 区域分配在 EP 控制器 2 的出站 (OB) 地址空间中。门铃和内存窗口 1 的 BAR 的分配和配置是在 NTB 端点功能驱动程序的初始化阶段完成的。从 EP 控制器 2 OB 空间到 PCI 地址空间的映射是在 HOST2 发送 CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL 时完成的。
9.6.2.3. 建模可选内存窗口：¶
此模型的建模方式与 MW1 相同，但每个额外的内存窗口都映射到单独的 BAR。