Apparently I have a combination of a D-Link switch, RTL8111 network card and a dubious network cable, that reliably crashes the FreeBSD driver on high usage. Turns out with either another cable or switch it works fine. Took me two years to figure out all my problems go away with a new 6€ cable.