Two ideas are powerful: non-uniform matching (adjusting the weights of different feature dimensions), and template adjustment (adjusting the templates themselves).
Both these ideas require a modified recognizer. Conceptually the recognizer must be designed with at least two (and preferably three) inputs: (1) the features, (2) the confidence map, (3) the template adjustment factors.