The rise of Generative AI is transforming how businesses build products, automate workflows, and deliver intelligent user experiences. From Large Language Models (LLMs) and AI Agents to Retrieval-Augmented Generation (RAG) pipelines and multi-agent systems, organizations are rapidly adopting AI-powered solutions to solve complex real-world challenges.
If you’re passionate about testing cutting-edge technology and want to help shape the future of AI reliability, Nimbusnext is looking for talented professionals to join its growing team as a QA Engineer โ Generative AI Systems.
๐ค About the Opportunity
At Nimbusnext, we’re building production-grade Generative AI applications powered by advanced Large Language Models, AI agents, multi-agent orchestration frameworks, Retrieval-Augmented Generation (RAG) pipelines, and AI-driven APIs.
This is a unique opportunity to work at the forefront of AI innovation while helping define the future of AI Quality Engineering.
๐ Position Details
Role: QA Engineer โ Generative AI Systems
Company: Nimbusnext
Location: Pune, Maharashtra
Work Mode: Hybrid / Onsite
Experience: 2โ5 Years
Industry: Generative AI, Artificial Intelligence, Software Engineering
๐ Why This Role Matters
Unlike traditional software applications, AI systems are non-deterministic. The same prompt can produce different outputs, agents can make unexpected decisions, and complex workflows can introduce new challenges around reliability, safety, and consistency.
Your work will directly influence the quality, trustworthiness, and safety of next-generation AI products used by customers and businesses.
๐ฏ Key Responsibilities
๐ค AI & System Testing
You will design and execute comprehensive testing strategies for AI-powered products and workflows.
โ
Test LLM-powered applications and AI-driven systems
โ
Validate AI agent behavior and multi-agent orchestration workflows
โ
Perform functional, integration, regression, exploratory, and black-box testing
โ
Evaluate Retrieval-Augmented Generation (RAG) pipelines
โ
Test prompts using variations, ambiguous inputs, and adversarial scenarios
โ
Detect hallucinations, inconsistencies, and unexpected AI behavior
โ
Validate response accuracy, latency, reliability, and error handling
โ
Conduct performance and stress testing for AI APIs and services
โ
Collaborate closely with AI engineers and developers to resolve issues
๐งช Test Design & Automation
Develop scalable testing approaches tailored to modern AI systems.
โ
Design structured test cases for prompt variations and context limits
โ
Validate multi-turn conversations and conversational memory
โ
Test tool-calling and function-calling workflows
โ
Verify structured outputs such as JSON schemas and API responses
โ
Develop automated regression suites for AI applications and APIs
โ
Support benchmark creation and evaluation frameworks
โ
Contribute to AI red-teaming and adversarial testing initiatives
๐ API & Integration Testing
Ensure reliable communication between AI services and supporting systems.
โ
Validate REST APIs and AI service integrations
โ
Test streaming responses and token efficiency
โ
Verify compatibility across model providers and deployment environments
โ
Ensure smooth integration between web applications, AI pipelines, and microservices
โ
Monitor API performance, scalability, and fault tolerance
๐ก๏ธ Safety, Security & Quality Monitoring
Generative AI systems introduce unique safety challenges that require continuous validation.
โ
Test for prompt injection vulnerabilities
โ
Identify potential data leakage risks
โ
Evaluate fairness, bias, and responsible AI behavior
โ
Support AI governance and compliance initiatives
โ
Define and monitor QA metrics for AI reliability and performance
๐ป Required Skills & Qualifications
The ideal candidate should possess a combination of software testing expertise and AI-focused quality engineering knowledge.
Core Requirements
โ 2โ5 years of QA experience
โ Experience testing web applications and APIs
โ Exposure to AI/ML-powered systems, chatbots, NLP applications, or LLM-based products
โ Strong understanding of REST APIs and JSON validation
โ Experience with automated testing frameworks
โ Knowledge of load testing and performance testing methodologies
โ Proficiency in Python or similar scripting languages
โ Understanding of prompt engineering concepts and LLM behavior
โ Strong analytical and problem-solving abilities
Preferred Skills
โญ Experience with RAG pipelines and vector databases
โญ Familiarity with AI agent frameworks and orchestration systems
โญ Knowledge of AI evaluation methodologies and benchmarking techniques
โญ Exposure to A/B testing for AI models
โญ Experience with AI safety and red-teaming frameworks
โญ Familiarity with OpenAI and other LLM APIs
โญ Understanding of CI/CD pipelines and modern release processes
โญ Experience working in fast-paced startup or product environments
๐ Success Metrics
Success in this role will be measured through meaningful business and engineering outcomes, including:
๐ Reduction in production AI failures and hallucinations
๐ Improved benchmark accuracy and evaluation stability
๐ Higher API reliability and system performance
๐ Enhanced AI safety and governance compliance
๐ Comprehensive documentation of limitations, risks, and edge cases
๐ฉ How to Apply
๐ฉ If interested, DM to Bhawana J. or send your resume to bhawana.joshi@nimbusnext.com
๐ฎ Shape the Future of AI Quality Engineering
The future of software testing is evolving alongside artificial intelligence. As organizations increasingly rely on AI-powered systems, the need for engineers who can validate, challenge, and improve these systems has never been greater.
If you’re passionate about breaking complex AI systems before users do, uncovering edge cases others miss, and building confidence in the next generation of AI products, then this role at Nimbusnext could be your perfect next career move.
