It may not be racist, per se, but it is prejudiced. It pre-judges people based upon their role as a member of a group, rather than their strengths and weaknesses as an individual. Affirmative action seeks to solve a problem (I won't hire you because you're this race/sex/ethnicity/sexual orientation) by applying the same problem in reverse (I have to hire a certain number of this race/sex/ethnicity/orientation). This does not result in more equality - at best, it's a break even, and at worst, it could result in the hiring of people unqualified to hold a position, just to meet a quota.
Affirmative action is a band-aid on top of a festering infection - it doesn't solve the problem, it just hides it for a while. Discrimination will always be present, and forcing those who would not hire (or promote, or admit, etc) a black man (for instance) to do so won't change that fact. In fact, it makes it worse - that person then carries with them the stigma of being hired/promoted/admitted within a system that uses affirmative action, and everyone, including himself, will wonder if he actually earned it or if he was just one of a number.