Three former presidents reportedly declined to attend President-elect Donald Trump's inaugural luncheon in a break from expected norms.