Guide: Setting Up an RDMA Cluster for AMD Strix Halo

This documentation provides a comprehensive walkthrough for configuring a dual-node cluster powered by AMD Strix Halo processors. The nodes are interconnected via Intel E810 (RoCE v2) network cards to enable distributed vLLM inference utilizing Tensor Parallelism (TP).

---

🚀 TL;DR (Quick Start)

If you are experienced, follow this checklist on both nodes:

• System Prep: Install Fedora 43 and Intel E810 NICs. Verify firmware via ethtool -i <interface>.
• Firmware/Kernel: Set iGPU to $512\text{MB}$ in BIOS. Add kernel parameters: iommu=pt and pci=realloc.
• Networking: Configure static IPs (192.168.100.1 and .2), set MTU to 9000, and configure firewall trust.
• Toolbox: Execute ./refresh_toolbox.sh to deploy the RDMA-enabled container and apply the librccl.so patch.
• Deployment: Export your HF_TOKEN (for gated models) and launch.

Pro Tip: The refresh_toolbox.sh script is intelligent; it automatically identifies your RDMA/Infiniband hardware and maps the devices into the container.

---

🧠 Core Concepts & Architecture

To optimize the Strix Halo cluster, it is essential to understand the underlying stack:

1. vLLM & Ray

vLLM serves as the high-performance inference engine. When a model is too large for a single APU's memory, vLLM employs Tensor Parallelism (TP) to shard the model across multiple devices. Ray is used as the orchestration layer to manage worker processes across the cluster.

2. RCCL (ROCm Collective Communication Library)

RCCL is the AMD counterpart to NVIDIA's NCCL. It manages the data plane, ensuring that tensor data is synchronized rapidly between GPUs. In a $\text{TP}=2$ configuration, nodes must exchange partial results after every single neural network layer.

3. RoCE v2 (RDMA over Converged Ethernet)

This protocol allows RCCL to perform Remote Direct Memory Access, writing data directly from one node's RAM to another's. By bypassing the CPU and the OS kernel, it drastically reduces latency.

System Flow Diagram

---

🛠️ Hardware Requirements

PCIe Connectivity Note: The Framework motherboard's PCIe slot is physically $\times 4$. To use a $\times 16$ card, you must use a riser (e.g., CY PCI-E $\times 4$ to $\times 16$ Extender). Standard slots are used $\rightarrow$ Note: One test board had its slot physically modified via an ultrasonic knife by Framework to fit $\times 16$ cards.

---

💻 Host Configuration (Fedora)

Perform these steps on the Host OS (Fedora 43) for both machines.

Verified Environment

Note: In-kernel drivers are sufficient; proprietary Intel drivers are not required.

• Ethernet Driver: ice
• RDMA Driver: irdma

4.1 Package Installation

Install the necessary RDMA userspace tools and benchmarks:

sudo dnf install rdma-core libibverbs-utils perftest

• rdma-core: Core libraries and daemons.
• libibverbs-utils: Tools like ibv_devinfo for device querying.
• perftest: Benchmarking tools (e.g., ib_write_bw).

4.2 Firmware Verification

Check your Intel E810 firmware version: ethtool -i enp194s0np0

Requirement: Firmware must be $\ge$ 4.91. If outdated, use the Intel® Ethernet NVM Update Tool.

Example Output:

driver: ice 
version: 6.18.5-200.fc43.x86_64 
firmware-version: 4.91 0x800214b5 1.3909.0 
bus-info: 0000:c2:00.0 
supports-statistics: yes 

4.3 Network Setup

We utilize the subnet 192.168.100.0/30.

Node 1 (Head):

sudo ip link set enp194s0np0 up
sudo ip addr add 192.168.100.1/30 dev enp194s0np0
sudo nmcli connection modify "rdma0" ethernet.mtu 9000
sudo nmcli connection up "rdma0"

Node 2 (Worker):

sudo ip link set enp194s0np0 up
sudo ip addr add 192.168.100.2/30 dev enp194s0np0
# Repeat MTU 9000 settings as above

4.4 BIOS & Kernel Tuning

1. BIOS: Set the iGPU memory allocation to 512MB.
2. Kernel: Apply the following parameters to your bootloader:
   • iommu=pt
   • pci=realloc

4.5 Firewall

Ensure the RDMA interface is trusted to prevent packet drops: sudo firewall-cmd --permanent --zone=trusted --add-interface=enp194s0np0

---

📦 Toolbox Installation

5.1 SSH Configuration

Before proceeding, establish passwordless SSH between Node 1 and Node 2 to allow Ray to orchestrate workers without manual intervention.

5.2 Installation Process

Run the provided setup script:

./refresh_toolbox.sh

This script handles:

1. Container deployment with RDMA support.
2. Application of the custom librccl.so patch.

5.3 RDMA Validation

Verify the connection is active using the following tools:

• Device Info: ibv_devinfo
• Bandwidth Test: ib_write_bw
• Latency Test: ib_send_lat

---

🏃 Running the Cluster

6.1 Verification

Before launching the model, run the setup verification suite to ensure the network and RCCL are communicating correctly.

6.2 vLLM Launch

Once verified, launch vLLM. Remember to provide your HuggingFace token for gated models: export HF_TOKEN=your_token_here

---

🛠️ Troubleshooting & References

• Issue: RDMA device not found in container. $\rightarrow$ Fix: Re-run refresh_toolbox.sh.
• Issue: Low throughput. $\rightarrow$ Fix: Double-check that MTU is set to 9000 on both nodes.

Acknowledgements: Special thanks to the Framework community and the vLLM contributors.