Software Engineer – AI Training & Model Evaluation

Software Engineer – AI Training & Model Evaluation Compensation: $60–$120/hour equivalent Location: Global | Fully Remote Employment Type: Contract Time Commitment: 15+ hours per week Schedule: Flexible About the Role We are seeking experienced Software Engineers with prior professional experience in AI training, model evaluation, or AI benchmarking to contribute to a project focused on improving advanced AI systems. In this role, you will leverage your software engineering expertise to design challenging programming tasks, evaluate AI-generated code, identify technical weaknesses, and establish objective standards for high-quality solutions. This opportunity is best suited for engineers who have previously worked on substantive AI-training projects and have a strong foundation in practical software development. Key Responsibilities • Design realistic and challenging software engineering tasks to evaluate advanced AI models. • Develop detailed task instructions, reproducible environments, reference solutions, and automated tests. • Review and evaluate AI-generated code for correctness, robustness, efficiency, and completeness. • Identify technical issues related to implementation, debugging, dependencies, performance, and reproducibility. • Create objective evaluation criteria and grading standards for technical submissions. • Assess model performance across different programming and software engineering scenarios. Required Qualifications • Prior paid professional experience in AI training, model evaluation, benchmarking, or improving AI systems. • Strong hands-on proficiency in Python. • Professional software engineering experience with at least one additional programming language, such as Java, JavaScript, TypeScript, Rust, C++, C#, or Go. • Strong understanding of software design, debugging, testing, version control, and dependency management. • Ability to assess complex codebases and identify technical issues independently. Public Proof of Technical Skills Applicants should be able to demonstrate their technical expertise through publicly accessible evidence, such as: • Substantial GitHub or GitLab repositories. • Meaningful open-source contributions. • Maintainer or core-contributor experience. • Competitive programming achievements (Codeforces, ICPC, Olympiads, etc.). • Publicly developed tools, libraries, or software systems. • Technical publications, benchmark contributions, or engineering write-ups. Preferred Qualifications • Experience creating AI-training tasks, reference solutions, automated tests, graders, or benchmark problems. • Experience reviewing and evaluating technical submissions from other AI trainers. • Codeforces rating of approximately 1900+ or equivalent competitive programming achievements. • Familiarity with Linux, Docker, CI/CD, command-line tools, and sandboxed development environments. • Experience with long-horizon or agentic coding evaluations. • Availability to start immediately.

Prior paid professional experience in AI training, model evaluation, benchmarking, or improving AI systems; strong hands-on proficiency in Python; professional software engineering experience with at least one additional programming language (e.g., Java, JavaScript, TypeScript, Rust, C++, C#, or Go); strong understanding of software design, debugging, testing, version control, and dependency management; ability to assess complex codebases and identify technical issues independently.

Design realistic and challenging software engineering tasks to evaluate advanced AI models; develop detailed task instructions, reproducible environments, reference solutions, and automated tests; review and evaluate AI-generated code for correctness, robustness, efficiency, and completeness; identify technical issues related to implementation, debugging, dependencies, performance, and reproducibility; create objective evaluation criteria and grading standards for technical submissions; assess model performance across different programming and software engineering scenarios.