HPE ProLiant DL360 Gen11 Networking

Networking Overview

The HPE ProLiant DL360 Gen11 does not provide embedded motherboard networking. Networking is configured using an OCP3.0 network adapter or a PCIe stand-up network adapter, with additional adapters supported depending on the server configuration.

The DL360 Gen11 supports Ethernet options ranging from 1GbE through 200GbE, as well as high-speed InfiniBand and Ethernet/InfiniBand adapters. The system includes two OCP slots on the motherboard, allowing OCP adapters to be installed without consuming standard PCIe expansion slots.

Ethernet PCIe Adapters

The following PCIe Ethernet adapters are listed in the DL360 Gen11 QuickSpecs. Available interfaces include BASE-T copper, SFP+, SFP28, QSFP28, QSFP56, and QSFP112.

PCIe Adapter Configuration Notes

• P08443-B21 uses PCIe 4.0 x8 and supports HH or LP form factors.
• P08458-B21 uses PCIe 4.0 x16 and supports FH/HL. If the secondary FH riser is not installed, a maximum of one FH PCIe card can be selected.
• P21106-B21 and P51178-B21 are 4-port 1GbE cards. A maximum of one 4-port card can be selected without the secondary riser and two with the secondary riser. These cards cannot be installed in PCIe Slot 2.
• P26253-B21, P26259-B21, and P26262-B21 use PCIe 3.0 x8.
• P87940-B21 uses PCIe 5.0 x8.
• P21109-B21 uses PCIe 3.0 x8.
• P26264-B21 uses PCIe 4.0 x16.
• P42044-B21 uses PCIe 4.0 x8.

Ethernet OCP3 Adapters

The DL360 Gen11 provides two OCP slots on the motherboard. OCP networking adapters do not consume the server's standard PCIe expansion slots. OCP Slot 2 is the preferred networking slot because it provides Share NIC functionality, including Wake-on-LAN.

OCP3 Bandwidth and Lane Configuration

Some DL360 Gen11 OCP3 adapters have a physical x16 design but can operate through an x8 connection. HPE specifically documents this for several adapters.

HPE notes that the x8 PCIe connection provides sufficient bandwidth for these adapters. A Gen5 x16 OCP3 cable connection can also be used with adapters physically designed for x16 connectivity.

Default OCP Networking Configuration

The Broadcom BCM5719 Ethernet 1Gb 4-port BASE-T OCP3 Adapter P51181-B21 is the OCP networking adapter defaulted by the configurator. Customers can remove it and select another PCIe or OCP card from the Networking, InfiniBand, Smart IO, or Storage Offload categories.

OCP Slot 2 is the primary OCP networking location because it provides Share NIC and Wake-on-LAN functionality.

OCP3 Enablement Kits

OCP slot connectivity depends on processor count, chassis configuration, and the selected OCP enablement kits.

OCP Enablement Rules

• OCP Slot 2 is the preferred networking slot because it supports Share NIC and Wake-on-LAN.
• In a one-processor configuration, P51911-B21 is defaulted when the BCM5719 OCP NIC is selected in OCP Slot 2.
• In a two-processor configuration, P48830-B21 or P48828-B21 is required when an OCP NIC is installed in OCP Slot 2.
• P48828-B21 and P48830-B21 require two processors and cannot be selected together.
• P51911-B21 cannot be selected in a two-processor configuration.
• P51911-B21 and P48827-B21 cannot be selected together in a one-processor configuration.
• If no OCP enablement kit is selected, an x16 OCP adapter cannot be selected and a maximum of one OCP/OROC device can be selected.
• If either the OCP1 x16 or OCP2 x16 enablement kit is selected, a maximum of one x16 100G-or-faster card can be selected. If both x16 enablement kits are selected, a maximum of two such cards can be selected.

20 EDSFF OCP Restrictions

The 20 EDSFF CTO model provides only x8 PCIe 5.0 electrical lanes to each of its two OCP slots. P48830-B21 can be used to enable OCP Slot 2. The P48827-B21 OCP1 x16, P48828-B21 OCP2 x16, and P51911-B21 CPU1-to-OCP2 x8 kits are not supported because the associated PCIe connectivity is used by the front storage configuration.

OCP Slot Priority

• Install an OCP networking card in Slot 2 first when possible because Slot 2 provides Share NIC and Wake-on-LAN.
• When two OCP networking cards are installed, the higher-speed NIC receives priority for OCP Slot 2.
• If OCP Slot 2 does not have an enablement kit, the slot is unavailable.
• OCP Slot 1 provides x8 electrical PCIe lanes by default from the motherboard when no OCP Slot 1 upgrade kit is installed.

InfiniBand and High-Speed Ethernet/InfiniBand Adapters

The DL360 Gen11 supports high-speed InfiniBand and dual-protocol Ethernet/InfiniBand adapters for HPC and other high-bandwidth workloads.

HPE identifies P45641-H24, P65333-H21, and P45642-H23 as transition-equivalent SKUs for earlier lifecycle SKUs. The NDR adapters use PCIe 5.0 x16. High-speed InfiniBand configurations have specific cooling, CPU, memory, storage, and ambient-temperature requirements.

Storage Offload Networking

The QuickSpecs also list network adapters designed for NVMe over Fabrics and storage-offload workloads.

NVIDIA 100Gb NVMe-oF Offload Adapter Requirements

The R8M41A adapter uses PCIe 3.0 x16. HPE requires a High Performance Fan Kit and secondary full-height riser for this adapter. Each storage-offload card also requires two supported 100GbE transceivers or cables. The QuickSpecs identify Q8J73A, Q2F19A, and Q9S71A as supported selections.

100GbE and Faster Networking Restrictions

High-speed network adapters require additional attention to system cooling and component selection. HPE specifies several important restrictions for 100GbE and faster networking and InfiniBand adapters.

• A High Performance Fan Kit is required for high-speed 100GbE-class and faster adapters covered by the QuickSpecs thermal matrices.
• The Standard Fan Kit cannot be configured with the identified 100GbE networking PCIe or OCP adapters.
• Performance Heatsinks, Closed Loop Liquid Cooling heatsinks, or a Direct Liquid Cooling module may be required depending on the configuration.
• If 256 GB memory is selected, high-speed Networking/InfiniBand PCIe or OCP adapters at 100G or higher generally cannot be selected without the specifically documented DLC exception.
• With DLC, some high-speed adapter configurations support 256 GB DIMMs at 1 DIMM per channel, up to eight DIMMs per socket.
• If a 500W power supply is selected, x16 Networking/InfiniBand adapters cannot be selected.
• Cooling requirements depend on processor TDP, chassis configuration, storage configuration, cable type, and ambient temperature.

High-Speed Networking and Storage

Some high-speed networking configurations reduce the supported storage configuration. HPE documents different limits based on whether the server uses 4 LFF, 8 SFF, 8+2 SFF, or 20 EDSFF storage, as well as whether the drives are SAS/SATA or NVMe.

With certain high-speed InfiniBand adapters and Closed Loop Liquid Cooling, SFF NVMe or EDSFF drives cannot be selected. For an InfiniBand NDR 1-port OSFP adapter in an 8 SFF CTO system with Closed Loop Liquid Cooling, HPE limits the configuration to a maximum of eight NVMe drives. For an InfiniBand NDR200 1-port QSFP adapter in a 20 EDSFF CTO system with Closed Loop Liquid Cooling, EDSFF drives cannot be selected.

DAC vs. AOC Thermal Considerations

The QuickSpecs thermal tables distinguish between Direct Attach Copper (DAC) and Active Optical Cable (AOC) configurations. In several high-speed networking configurations, DAC cabling allows a higher supported ambient temperature than AOC. Exact limits vary by adapter, processor TDP, cooling solution, and drive configuration.