Intelligent Test Suite Selector (ITSS)
Optimizing Testing Strategies Through Code Change Analysis
Project Overview
Developed an AI-powered solution that analyzes code changes and recommends optimal testing strategies, reducing unnecessary testing costs by 35% while maintaining 98% defect detection coverage. The system provides visualizations of how changes impact core application components.
Problem
Enterprise software teams were spending 40% of their CI/CD budget on unnecessary tests, while still missing critical test scenarios. Manual test selection led to either over-testing (wasting resources) or under-testing (increasing production defects). Analysis showed teams lacked visibility into how code changes actually impacted system components.
Solution
Created an intelligent system that: 1) Analyzes code changes line-by-line, 2) Maps dependencies between components, 3) Classifies impact (UI, database, etc.)
Tools
- Python
- React
- AWS Lambda
- GitHub API
- D3.js
- PyTorch
- Docker
Timeline
- Requirements Gathering: 2 weeks
- Architecture Design: 3 weeks
- Core Engine Development: 6 weeks
- UI/Visualization Development: 4 weeks
- Pilot Testing & Refinement: 3 weeks
Team
- 1 Product Manager (me)
- 2 Machine Learning Engineers
- 3 Full-Stack Developers
- 1 UX Designer
- 2 Quality Assurance Engineers
My Role
- Led product vision and roadmap development
- Conducted stakeholder interviews with DevOps teams
- Designed the change impact classification algorithm
- Spearheaded the visualization approach for technical audiences
- Coordinated between ML and frontend teams
- Presented results to CTO and engineering leadership
My Design Process
Problem Validation
Interviewed 15+ engineering teams to understand pain points
Data Analysis
Analyzed historical test runs and defect reports
Algorithm Design
Developed change classification and impact scoring models
Visualization Prototyping
Created interactive dependency graph prototypes
Integration Development
Built GitHub integration and CI/CD plugins
Pilot Testing
Deployed to 3 teams for real-world validation
Process Documentation
Mapping different iterations of the website to improve usability and functionality
Research
Methods
- Engineering team interviews
- CI/CD pipeline analysis
- Test coverage audits
- Defect root cause analysis
- Prototype usability testing
Key Insights
- 82% of engineers wanted better visibility into change impact
- Test suites were often selected based on habit rather than change analysis
- Dependency mapping was the most requested feature by architects
- Teams needed both automated recommendations and explanatory visualizations
Final Designs
Detailed view of how a specific commit impacts components
Overview of testing analysis
Component impact analysis showing which tests are affected by code changes
Dependency Graph visualizing how components are interconnected and affected by changes
Outcomes
- 35% reduction in unnecessary testing costs
- 98% critical defect detection rate maintained
- 40% faster test selection decisions
- Adopted by 15+ engineering teams within 3 months