Linux drivers for communicating with the FLX-182 card through a virtual network over PCIe

These modules creates a network device on Versal card and host PC. Each FLX-182 card connected to PCIe becomes a peer in the network and is accessible from the host. FLX-182 peers cannot access each other. Host device is assigned a random MAC address on startup. FLX-182 devices are assigned a MAC address generated from IDCODE and VERSION information, so that it stays persistent between reboots.

Instructions

• Instantiate the versal_network_device from https://gitlab.cern.ch/atlas-tdaq-felix/firmware/-/tree/phase2/FLX-1886_FLX182_virtual_network/sources/ip_cores/BNL182/ip_repo/versal_network_device_1.0?ref_type=heads

• Connect the AXI bus to the Versal processing system

• Connect the FIFO ports to the the corresponding ports of Wupper

• The following modules need to be cross-compiled with petalinux for the Versal card

• Run petalinux-build -C flxnet in the petalinux project to build the correct modules:

  • flxnet_dev.ko
  • flxnet_target.ko

• Copy Makefile.host to Makefile on host PC and run make to build the correct modules for the host PC:

  • flxnet_dev.ko
  • flxnet_pcie.ko

• Add the following device tree entry in the petalinux project (system-user.dtsi file in flx-petalinux/project-spec/meta-user/recipes-bsp/device-tree/files):

  &amba {
      flxnet_instance: flxnet@20100030000 {
          #address-cells = <0x02>;
          #size-cells = <0x02>;
          reg = <0x201 0x00030000 0x00 0x100>;        
          compatible = "atlas_tdaq_felix,VERSAL_virtual_network_device_v1.0";
          status = "okay";
      };
  };

  • The address (the number to the right of '@' and two first numbers in reg field) must match the Vivado address editor for the VERSAL_virtual_network IP

• Setup a DHCP server on the host to automatically configure an IP address on the Versal cards

• For debugging messages, increase the message level of the ethernet adapter using the commands:

  ethtool -s flxnet0 msglvl 4095  # display all debug messages
  ethtool -s flxnet0 msglvl 8191  # display all debug messages and packet data

  • The exact message level may be different for different Linux versions
  • For detailed explanation debug message levels see: https://www.kernel.org/doc/Documentation/networking/netif-msg.txt

The modules

• flxnet_dev.ko : network interface representation for the FIFOs, shared module between the Versal and the host, it exports a function that the other modules use for registering the device as a network peer
• flxnet_target.ko : driver for the Versal card, it obtains the IP information from the device tree, then adds the peer to the network
• flxnet_pcie.ko : driver for the host PC, it finds the correct PCIe devices and adds them as peers into the network

Update

Readme updated on 13-11-2024 by Laura Rientsma l.rientsma@nikhef.nl
Original author of the driver is Elena Zhivun, last edited on 05-07-2022
For questions, please contact Frans Schreuder f.schreuder@nikhef.nl

Biggest changes:

• Driver is adjusted for and tested with the FLX-182 card instead of the VMK180
• Driver is made compatible with modern Linux designs
• Changed FIFOs into AXI Stream FIFOs, added end of packet signal that fixed the delay of the first few packages
• The driver is written so that multiple flxnet network devices can be created and communicate with one host
• DHCP server on host automatically gives out an IP address to the Versal card instead of having to configure the IP addresses manually
• Added flxnet driver to the build_rpm.sh script from https://gitlab.cern.ch/atlas-tdaq-felix/felix-tdaq-drivers/-/tree/main?ref_type=heads, so that the modules on the host PC automatically insert on host server startup
• Added carrier function that gives a signal when the link goes up or down