The prefetch memory region of an NTB adapter is used for mapping remote segments and multicast memory groups. When a remote segment with size 32MB is mapped, 32MB of the adapters prefetch space is consumed until the segment is unmapped. Most of the prefetch memory address range is available for customer use, but is a shared resource for all concurrent use of PCIe. eXpressWare will typically use a small portion of this resource for internal communication and protocols.
The factory default prefetch memory size of the Dolphin adapters is 256MB.
Due to various hardware constrains such as alignment requirements the full prefetch memory space might not be usable. The prefetch memory space size can be increased if needed.
The currently configured PCIe prefetch memory size is typically displayed by dis_diag and can be queried by running dis_config:
[root@PXMXS-01 ~]# dis_config -gpms Card 1 - Prefetch memory space size: 512
The PCIe card typically support a very large address space, but your BIOS will in many cases limit this significantly. The BIOS will in many cases refuse to boot the system so you have to be very careful to test the desired size on one system before you do this on all systems.
If your system refuses to boot after the change, you have to modify the DIP switch to select another NTB setting with a smaller prefetch memory size.
If you intend to increase the PCIe Prefetch Memory size beyond the factory default, it is strongly recommended to ensure that 4G decoding is enabled in the system BIOS before proceeding. This is sometimes also referred to as “64-bit BAR support” “64-bit address decoding”. Consult your system or motherboard manual for details. Some systems may fail to boot if the prefetch memory size is increased beyond the factory default. To recover from this the prefetch memory size must be decreased.
The PCIe Prefetch Memory size is changed by modifying the DIP switch setting. Please consult the MXH930, MXH940, MXH950 or MXH830 NTB card Users Guides for more information.