In the weekend I added support for the Marker protocol in lagg(4).
The basic function of it is to verify that the interface queues are empty for all the aggregated interfaces. Frame distribution is supressed and a marker frame places on the queue for each interface, the frame will be sent to the partner device and then returned immediately on the same link. Once each frame is received then we can be sure that no other ethernet frames are queued for that port and when all ports are complete then frame distribution is re-enabled.
All this is needed because frames must be delivered in order for Ethernet, the lagg device can send frames over different ports to utilize all the combined bandwidth by hashing the header (ethernet src/dst, vlan tag, ip header, ipv6 header). The hash selects the transmit port and this ensures for each connection the frames take exactly the same path. If a port is added or removed then the hash output is affected since its modulo the number of ports, the marker protocol ensures that frames for a particular connection are not put on a different interface queue and arrive out of order.
My next job is to add bgscan for iwi(4) and hopefully the card will arrive tomorrow.