A reliable identity management system is urgently needed in order to combat the epidemic growth in identity theft and to meet the increased security requirements in a variety of applications. Biometric authentication establishes the identity of a person based on “who he is", rather than “what he knows" or “what he possess”. There are usually two very serious criticisms against biometric technology that have not been addressed satisfactorily, biometrics are not secrets and enrolled biometric templates are not revocable. The first fact implies that the attacker has a ready access to the legitimate biometric trait (e.g., facial image finger print of an enrolled user of the biometric system) and, therefore, could fraudulently inject it into the biometric system to gain access. The second fact implies that when a biometric trait has been “compromised”, the legitimate user has no recourse to revoking the trait. In this work, we devise a technique that complements the weakness of a biometric system by Combining both biometrics and a Visual–based challenge response for access control, thus Enhancing Biometric System Using Visual-Based Challenge Response, having considered human factor.