• LLM performance across languages: Most LLMs are trained on corpora with multiple languages but show differential performance. Popular training datasets don't represent languages like Rust well. This post shares early results on improving LLM performance for code completion in such languages.
• Cody's autocomplete process: Developers use Cody for code completion in IDEs. Cody performs a loop per minute looking at context, stitching prompts, triggering completions, post-processing LLM output, and displaying suggestions. The lifecycle of an autocomplete request is described in detail.
• Evaluation benchmark: The HumanEval benchmark measures the functional correctness of programs generated by LLMs. The pass@k metric assesses the top k-generated code samples passing unit tests. Popular LLMs show differences in performance across languages like Python, Rust, Ruby, and Matlab.
• Latency vs. Pass@1: In code completion, latency (response time) and Pass@1 (accuracy) are important metrics. GPT4-turbo and Claude-3-opus have good Pass@1 but high latency. Models within 1000ms latency like mixtral-8x7b, codellama-34b-instruct, and starcoder-16b are promising.
• Fine-tuning for Rust: Rust has unique challenges for code completion due to safety constraints. LoRA is used for fine-tuning Mixtral 8x7b and Code Llama 34b on Rust datasets. Finetuned Mixtral shows significant improvement in Pass@1 metric.
• Concrete examples: Five examples show the effectiveness of the finetuned Mixtral model. It corrects errors in the base model's solutions for Rust code, such as handling method calls, immutability, and sorting. The finetuned model provides more robust and correct solutions.
• Looking ahead: Success with Rust is just the beginning. They will share progress on other languages, develop more representative evaluation suites, and roll out finetuned models. Invite users to try Cody and provide feedback.
• Use Sourcegraph: Sourcegraph can be used to industrialize software development. It offers a simple CTA.