If I'm not mistaken, раздражать/раздражить can also mean "to cause physical irritation" (for example, if you get a skin-rash from itchy woolen sweaters).
And a question for native Russians: what about the verbs досадовать or досадить -- can they be used here?