I've got a 4890 with fully up to date drivers. Tried reinstalling the game already. Tried turning everything graphical off and forcing vsync on in the ini and in the ATI Catalyst Control Center.
People suggested my graphics card was overheating. I dunno what's normal but according to the CCC it's idling at ~59C and after alt-tabbing out of Borderlands it was up to 73C at most. But I tried with Crysis and it showed over 70C when I alt-tabbed out of that but Crysis has no issues like this :(