I usually have a firearm on my person, or in my vehicle, so leaving keys (to the car, or to the lock-box) anywhere near it is a huge no-no.
Situations like this are what make the whole “that’s 100 grams lighter” conversation incredibly stupid to me… I keep my car keys (and basic bike tools) in a tail bag, under my seat. If the weight is having an adverse effect on my riding, I haven’t noticed it.