Introduction

In the rapidly evolving landscape of technology, System-on-Chip (SoC) designs have become increasingly complex and integral to a wide range of applications. As these designs grow in complexity, so do the security challenges associated with them. One critical aspect that demands attention is the security validation of Scalable I/O Virtualization (SIOV) at the SoC level.

Understanding Scalable I/O Virtualization

The scope of this article is not to delve into the architectural details of Scalable I/O Virtualization (S-IOV).  Instead we are to focus on security testing a SoC level with S-IOV virtualization.  Scalable I/O Virtualization is a technology that allows multiple virtual machines (VMs) to share a single physical I/O device efficiently. This technology is essential for improving the performance and scalability of virtualized environments. However, the integration of SIOV into SoC designs introduces new security vulnerabilities that must be addressed to ensure the overall security of the system..

The importance of Security Validation

1. Complexity and Interconnectivity: Modern SoC designs incorporate numerous third-party intellectual property (IP) blocks, each with its own set of security considerations. The interconnectivity between these IP blocks can create potential attack vectors that malicious actors can exploit. Security validation ensures that these connections are secure and that data integrity is maintained.

2. Third-Party IP Integration: The use of third-party IPs in SoC designs is common practice to meet time-to-market demands. However, these IPs may not always adhere to the same security standards as the rest of the SoC. Security validation helps identify and mitigate vulnerabilities introduced by third-party IPs.

3. Dynamic Nature of Virtualization: SIOV allows for dynamic allocation and reallocation of I/O resources to different VMs. This dynamic nature can lead to security gaps if not properly validated. Security validation ensures that the allocation process is secure and that unauthorized access to I/O resources is prevented.

4. Firmware and Software Vulnerabilities: In addition to hardware vulnerabilities, firmware and software running on the SoC can also introduce security risks. Security validation at the SoC level includes assessing the security of firmware and software components to ensure they do not compromise the overall security of the system.  We will focus on this aspect in the discussion ahead.

Host software in the role of exposing security vulnerabilities in Virtualization 

In a previous article we discussed the ways at Security ring layers that modern OS (such as Linux) provide least amount of privileges for the scope of their domain.  These are ring layers:

If we look at S-IOV, there are varying levels of privileges and restrictions placed upon Virtual Machines (VMs) and Host software

Figure 2 shows the aspects of S-IOV architecture including Guest Drivers, Host Drivers, the IOMMU and the PCIe endpoint (or device).  For the purpose of this article, we will reduce the complexity to that just of Host drivers with Ring 1 access, the IOMMU, and the PCIe endpoints under test.  The VMs in Figure 3 we can discuss in a future article.

Figure 4 shows that multiple host drivers with Ring 1 privileges can access both the config and MMIO space of the PCIe endpoints, i.e. the S-IOV devices.  They are enabled to do this by the IOMMU who sets up the page tables, in this case 4K page tables, such that they map to both the Physical and Provisioned device resources.  Usually provisioned to VMs but in our case we are just focusing on Host OS software.

The steps for the driver setup of this MMIO space is shown below

Figure 5 shows our specific test scenario that we want to explore.  This is where two different PCIe endpoints MMIO spaces are accessed when 4K page tables are setup such that one such page for Device B is adjacent to a page for Device A. Without intention and probably at random as that would be the in domain of the IOMMU.  That is, where it places the 4K pages within the system address space. In our example the flow occurs in this manner:

A)     Host Driver 1 programmatically has the S-IOV virtualization engine setup page tables.  These are usually done by assigning PASID such that there is granular address translations. 

B)     Host Driver 2 independently in a separate process also sets up the page tables.

C)    Device A and Device B now have 3 different pages mapping their MMIO space so that they can be addressed by their respective drivers.

D)    In our instance the 3rd page set up for Host Driver 1 is adjacent to the first page for Host Driver 2.  Setup such that it is seen by the Host OS as virtualized addresses.

This is an interesting scenario in that a non-posted read by Host driver 2 could in programming error or by malicious actor read the 0x(N-1) address instead of the top of the 4K page as it should.  As it has ring 1 privileges it could be allowed to attempt this access.  This would result in a page fault.

This non-posted read to MMIO space in x86 modern server systems should read at most 8 bytes.  It could be more depending on using more advanced x86 instructions.  The 4K pages are 64 byte aligned  However if Host driver 2 reads 0x(N-1) where again address 0xN is 4K and 64 byte line aligned then at least 4 bytes would be read.

Figure 6 shows the example of address 0x3F000000 as the start of Device B MMIO space.  So in our security test addresses 0x3EFFFFFC to 0x3EFFFFFF could be read in error by host driver 1.

Figure 7 illustrates what should happen in our example such that this read would result in an error and a correctable/uncorrectable AER error propagated to the PCIe root.

Test Flow

For our security testing the test flow should be as shown in Figure 7:

1)      Host Driver 1 and Driver 2 programmatically request IOMMU to setup page tables to access PCIe endpoints Device A and Device B

2)      IOMMU virtualization engine sets up page tables

3)      Host Driver 2 reads (in programming error or by malicious actor), for example 0x3EFFFFFF instead of 0x3F000000.

4)      Device A gets a non-posted read to its MMIO space, address 0x(N-1). This request should only come from Driver 1 and not Driver 2. Address 0x(N-1) could be protected space or MMIO space that has known issue to not be read.  This is the security vulnerability we are testing for.

5)      An AER error as well as a page fault should occur at this point.  The TLP should be received by the Host CPU with a UR (Unsupported Request) in the TLP header.

6)      Device A should report an AER correctable/uncorrectable error to the PCIe root port. 

7)      System should be handle to the UR and Page Fault gracefully but what if Device A MMIO space is protected or has some known hardware issue. Could result in unpredictable behavior

Trusted Software

Trusted software, such as host device drivers, should be trusted to not access MMIO space.  However the behavior of software errors should be examined when this occurs.  In addition, some MMIO space for PCIe endpoints maybe for functions not ready to be exposed for use.  MMIO space for Device A may cause unpredictable hardware behavior if it has been directed to not read from such space.

Another version of this test scenario is to “fall off” the edge of Device A’s page table by reading the last byte address of Device A and therefore at least four bytes in total, 3 of those in Device Bs page table. 

Variations of non-posted reads could be done.  Examples are both 4 and 8 byte MMIO reads.  Future platforms could have even more bytes read.

This test scenario could be created with the IOMMU creating page tables at random addressable space or restricting the IOMMU such that there are these adjacent 4K pages.       

Conclusion

This has just been an example of innovative validation.  Here At Approaching Zero Escapes Validation and Development, LLC, we strive to continue to increase verification and validation coverage.  And bring that new coverage to our clients.  You want zero bug escapes, to both internal customers and external customers.  At Approaching Zero Escapes, we want to partner with you to achieve that.

References

PCIe - TLP Header, Packet Formats, Address Translation, Config Space, Command Register, Configuration types

https://en.wikipedia.org/wiki/Protection_ring

OCP Scalable I/O Virtualization Specification Scalable I/O Virtualization – R1.0, Ver 1.3, June 2022