Overview

In SoC environments with a PCIe root and large PCIe endpoint trees, there is a need to add non-transparent bridges to expand address space for these addressable PCIe endpoints.  A local CPU PCIe root limits the tree to a common address space.  A non-transparent bridge address allows that address space to be expanded.  For example, this could include bridges between multiple platforms in a data center.    

For validation, this provides an opportunity to expand security testing such that a trusted agent, in this case a NTB bridge appearing as a local PCIe endpoint, be tested as if controlled by a malicious actor.  A malicious actor that could originate from a separate data center platform and target local protected memory.

 From a linux OS standpoint, a non-transparent bridge sees it as another PCIe endpoint in the system, with its own BDF (Base,Device,Function) and when booted has its own BAR and associated MMIO (Memory Mapped I/O) allocated space.  This endpoint should be given Ring 1 security privileges as shown below by the OS: 

However, if the platform has not booted to OS, but instead is at an EFI shell, then the BDF has been assigned to the PCIe endpoints, including the NTB.  As well as the BAR and MMIO space for the endpoints.  This is done by BIOS/p-code.  The OS has not been booted to prevent malicious access such as from device drivers.

The NTB bridge on the target system could be setup as shown below:

For example, the malicious agent on the remote platform could address VGA memory from 0xC0000 to 0xCFFFF on the target platform (Side B). 

When the malicious agent on the remote platform attempts to do a non posted read of the target VGA space, the read should have all 0s.  Likewise, a write across the NTB non-transparent bridge to VGA memory should not be allowed. 

Testing Methodology

Detailed test case definition and test case development is beyond the scope of this article.  However, by following this methodology the following could be tested.

1)      For all protected memory regions on target system such as in the DOS legacy address range,

a)      Setup side A and side B of the NTB.  At least on Side B with EFI shell

b)     On side B, setup the address translation register

c)      From Side A attempt non-posted read with a given offset to read protected memory.

d)     Ensure all 0s returned or CTO (Completion Timeout)

e)      Attempt a posted write to protected memory.   Verify on Side B that no overwrite occurred.

References

AN-707: IDT PCIe Gen2 Switch Family Non-Transparent Operations

Conclusion

This has just been an example of innovative security testing.  Here At Approaching Zero Escapes Validation and Development; LLC, we strive to continue to increase verification and validation coverage.  And bring that new coverage to our clients.  You want zero escapes, to both internal customers and external customers.  At Approaching Zero Escapes, we want to partner with you to achieve that.