We are looking for a detail-oriented and experienced Gen-AI QA Engineer to join our team. This role will focus on ensuring the quality and reliability of our natural language processing (NLP) and large language model (LLM) systems, including fine-tuned models, retrieval-augmented generation (RAG) workflows, and enterprise AI integrations
Responsibilities
- Develop and execute test plans and test cases for NLP and LLM systems, ensuring they meet quality and performance standards.
- Apply a strong understanding of AI/ML model testing methodologies, including performance evaluation, error analysis, hallucination detection, and fairness/bias evaluation.
- Design and implement test automation for AI/ML applications, ensuring repeatable and scalable validation processes.
- Utilize Python and relevant testing frameworks (e.g., pytest, unittest, Selenium) to automate functional and integration tests.
- Conduct adversarial testing to identify vulnerabilities and edge cases in AI models.
- Perform system integration testing, ensuring seamless interaction between AI models and enterprise applications like Salesforce and SAP.
- Validate AI-generated outputs for accuracy, consistency, and compliance with business logic.
- Ensure compliance with data privacy regulations, security protocols, and ethical considerations in AI.
- Collaborate with data scientists and engineers to identify and resolve issues, improve model quality, and refine testing workflows.
- Document and communicate test results, defects, and recommendations to stakeholders.
Requirements
Experience: 3+ years in QA testing, preferably in AI/ML applications.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Strong understanding of NLP, LLMs, fine-tuning, and retrieval-augmented generation (RAG).
- Proficiency in Python and relevant testing frameworks (pytest, unittest, Selenium, Robot Framework).
- Experience with ML testing methodologies, including performance benchmarking and error analysis.
- Familiarity with cloud platforms (AWS, GCP, Azure) and distributed systems testing.
- Knowledge of API testing, system integration testing, and experience working with enterprise applications like Salesforce and SAP.
- Understanding of automation testing for AI/ML pipelines.
- Strong analytical skills with attention to detail and problem-solving ability.
- Excellent communication skills to interact with technical teams and stakeholders.
Preferred Qualifications
- Experience with test automation tools and frameworks for AI/ML applications.
- Contributions to open-source QA projects or publications in the field of AI testing.
- Certifications in QA or software testing (e.g., ISTQB).
- Experience in an Agile development environment.
