I think it is accusative if it is used in the sense of "I am sorry for you". If it is used in the sense of "I don't want to give you away (grudge)" then it can be either accusative or genitive (мне жаль книгу/книги).