How to do a TRUE rescan of PCIe bus

Question

I have an FPGA (Like most of the people asking this question) that gets configured after my Linux kernel does the initial PCIe bus scan and enumeration. As you can guess, the FPGA implements a PCIe endpoint.

I would Like to have the PCIe core re-enumerate the ENTIRE PCIe bus so that my FPGA will then show up and I can load my driver module. I would also like the ability to SWAP the FPGA load out for a different configuration. By this I mean I would like to be able to:

1. Boot Linux
2. Configure FPGA
3. Enumerate PCIe endpoint and load module
4. Remove PCIe endpoint
5. Re-configure FPGA
6. Re-enumerate PCIe endpoint

All without rebooting Linux

Here are solutions that have been proposed elsewhere but do not solve the problem.

echo 1 > /sys/bus/pci/rescan This seems to work (only sometimes) and it does not work if I want to hotswap the FPGA load after it was first enumerated.

Can the Hotplug/power managment facilities of PCIe be used to make this work? If so is there any good resources for how to use the Hotplug system with PCIe? (LDD does not quite cover it thoroughly enough)

Answer 1

Re-enumerating the PCIe bus/tree via echo 1 > /sys/bus/pci/rescan is the correct solution. We are using it the same way as you described it.

We are using echo 1 > $pcidevice/remove to disconnect the driver from the device and to detach the device from the tree. The driver (xillybus) is not unloaded, just disconnected.

A better solution is to rescan only the node where your FPGA is attached to. This reduces the over all impact for the system.

This technique is used in the RC3E FPGA cloud system.