I assigned one pic to all contacts, I meant to only assign it to one.
There's no option to unassign a photo from all contacts at once. What I can suggest is to pick a picture you want to to have as your default picture for your current contacts. So rather than deleting, you're just replacing them.