Ok, I’m back. I did some quick research and it looks like that Mikrotik switch should be able to do line-rate between the SFP+ ports. That’s important because if it was somehow doing non-hardware switching, the performance would be awful. That said, my personal opinion is that Mikrotik products are rather unintuitive to use. My experience has been with older Ubiquiti gear and even older HP Procurve enterprise switches. To be fair, though, prosumer products like from Mikrotik have to make some tradeoffs compared to the money-is-no-object enterprise space. But I wasn’t thrilled with the CLI on their routers; maybe the switches are better?

Moving on, that NIC card appears to be equivalent to an Intel x520, so drivers and support should exist for any mainline OS you’re running. For 10 Gbps beyond, I agree that you want to go with pluggable modules when possible, unless you absolutely know that the installation will never run fibre.

I will note that 10 Gbps over Cat 5e – while not mentioned in the standard and thus officially undefined behavior – it has been reported to work over short distance, in the range of 15-30 meters by some accounts. The twisted pair Ethernet specs only call out the supported wire types by their category designation but ultimately, it’s the signal integrity of the differential signals that matter. Cat 3, 5, 5e, 6, etc are just increasingly better at maintaining a signal over a distance. This being officially undefined just means that if it doesn’t work, the manufacturer told no lie.

But you’re right to avoid 10 Gbps twisted pair, as the xcvrs are expensive, thermally ridiculous, power hungry, and themselves have length limits shorter than what the spec allows, because it’s hard to stuff all the hardware into an SFP+ pluggable module. Whereas -SR optics are cheap and DACs even cheaper (when the distance is short enough). No real reason to adopt twisted pair 10 Gbps if fibre is an option.

That said, I didn’t check the compatibility of your selected SR transceiver against your NICs and switch, so I’ll presume you’ve done your homework for that.

Going back to the x8 card in a electrically x4 slot, there’s a thing in the PCIe spec where the only two widths that are mandatory to support are: 1) the physical card width, and 2) the 1x width. No other widths are necessarily supported. So there’s a small possibility that the NIC will only connect at 1x PCIe, which will severely limit your performance. But this is kinda pathological and 9 out of 10 PCIe cards will do graceful width reduction, beyond what the PCIe spec demands. And being an x520 variant, I would expect the driver to have no issue with that, as crummy PCIe drivers can break when their bad assumptions fall through.

Overall, I don’t see any burning red flags with your plan. I hope you’ll update us with new posts as things progress!