Naw, it's easy. All you need is a 9/64 hex wrench. You can unscrew the old and screw on a new one in about a minute or so. I sometimes swap silver and black ones between hilts.
Sweet! I have a set with all sizes, so I should be fine with the key size. Just placed the order for my wheel. Glad to know it's actually as easy as it seemed XD