Solve the actual problem the key was solving
Most spare keys exist for a genuine reason: emergencies, plumbers, plants, a child who gets home before you do. If you take access away without replacing the function, you create a practical hole and hand the other person a legitimate argument.
So replace it first. A small combination lockbox by the door covers tradespeople and forgotten keys, and a code can be changed in seconds without anyone's feelings being involved. A neighbour who does not have an emotional stake in your marriage is often the better emergency keyholder than a relative who does. If someone is coming in to water plants while you travel, agree the days out loud rather than leaving it open-ended.
When you have a system, the conversation becomes almost boring: "We've set up a lockbox for emergencies, so we're collecting the spare keys back in." Boring is the goal. Boring is what stops it becoming a story told at Christmas for the next decade.
If someone has already used it badly
Address the specific incident quickly, while it is still small, and stay off the topic of character. "When you came in on Sunday while we were asleep, it really threw me" is answerable. "You have no respect for our privacy" is a declaration of war and will be defended accordingly.






