If the cpu port is connected through SGMII we need to enable SerDes for
it to work.

Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>
Acked-by: Felix Fietkau <nbd@nbd.name>