Improve handling of misaligned annotations