My switch box also has a DPDT toggle switch for headphone/speaker output (grounds are connected together). That's not a problem on the output, but a bad idea for the input side ...

Menlo Microsystems’ MM5600 DPDT differential switch offers high-speed signal switching from DC to 20 GHz or up to 40 Gbps. According to the manufacturer, the device outperforms conventional ...