You don't use the charging handle that often. In any reload situation, the bolts either locked back or behind a chambered round, neither situation requires it to be messed with.
The only time I use it is the clear a malfunction or the first shot in a string, I don't do either while trying to...