Any contacts you've interacted with in Clay, meaning added to a Group or Starred, taken a Note on, set a Reconnect cadence for, etc won't be removed since you've created data around them.
The easiest way to remove multiple people is by selecting multiple people in Search and choosing 'Archive' in the Command Bar.