The negotiations and alliance switching reached a fever pitch, the final bit of life was beaten out of the myriad running gags, the whole film was flooded with too many characters trying to be funny and failing (that brethren court scene was embarrassing), they finally overused Jack Sparrow with the thousand versions, etc., I could go on. I've seen the film quite a few times now, and on each subsequent re-watching, I like it slightly less than before. It just feels to me as if most of the actors are going through the motions to go through them. I also read that Gore Verbinksi was unhappy working on those last two films, due to issues like shooting without finished scripts and all that sort of thing, which definitely shows in the excessive, sprawling and often messy nature of the sequels.
Don't get me wrong, there was a lot to like in the Pirates movies, but the sequels definitely went downhill.