GitHub launched an AI-powered secret scanning feature within Copilot integrated into GitHub Secret Protection. It uses context analysis to improve password detection in code, addressing shortcomings of traditional methods. A blog post detailed its development, and a senior software engineer commented on its significance. During private preview, challenges included dealing with unconventional file types. GitHub enhanced its offline evaluation framework by incorporating feedback and using GPT-4 to generate test cases. They experimented with different techniques like using GPT-4 as a confirming scanner and MetaReflection technique. A "mirror testing" framework was implemented to validate improvements and assess impact. Testing showed a significant drop in detections and false positives with minimal impact on actual passwords. Lessons learned include prioritizing accuracy, using diverse test cases, managing resources, and collaborating. Copilot secret scanning is now part of security configurations.