chore(ci): remove deprecated CI Monitor workflow (#20993)
This commit is contained in:
Lianmin Zheng
@@ -1,334 +1,55 @@
|
||||
# SGLang CI Monitor
|
||||
# SGLang CI failure monitoring
|
||||
|
||||
> **Note**: This README.md is primarily generated by Claude 4 with some manual adjustments.
|
||||
Scripts used by [.github/workflows/ci-failure-monitor.yml](../../.github/workflows/ci-failure-monitor.yml): scheduled failure analysis and optional Slack notifications.
|
||||
|
||||
A comprehensive toolkit to analyze CI failures and performance trends for the SGLang project. This toolkit includes four main tools:
|
||||
## Tools
|
||||
|
||||
1. **CI Analyzer** (`ci_analyzer.py`): Analyzes CI failures and provides detailed failure pattern analysis
|
||||
2. **Performance Analyzer** (`ci_analyzer_perf.py`): Tracks performance metrics over time and generates trend charts
|
||||
3. **Test Balance Analyzer** (`ci_analyzer_balance.py`): Analyzes test time gaps between elapsed and estimated times to help balance CI
|
||||
4. **Failures Analyzer** (`ci_failures_analysis.py`): Tracks consecutive failures, identifies flaky jobs, and monitors runner health
|
||||
1. **Failures Analyzer** (`ci_failures_analysis.py`): Tracks consecutive failures, identifies flaky jobs, and monitors runner health across PR Test / Nightly workflows (Nvidia, AMD, Intel, XPU, NPU).
|
||||
|
||||
## Features
|
||||
|
||||
### CI Analyzer (`ci_analyzer.py`)
|
||||
- **Simple Analysis**: Analyze recent CI runs and identify failure patterns
|
||||
- **Category Classification**: Automatically categorize failures by type (unit-test, performance, etc.)
|
||||
- **Pattern Recognition**: Identify common failure patterns (timeouts, build failures, etc.)
|
||||
- **CI Links**: Direct links to recent failed CI runs for detailed investigation
|
||||
- **Last Success Tracking**: Track the last successful run for each failed job with PR information
|
||||
- **JSON Export**: Export detailed analysis data to JSON format
|
||||
|
||||
### Performance Analyzer (`ci_analyzer_perf.py`)
|
||||
- **Performance Tracking**: Monitor performance metrics across CI runs over time
|
||||
- **Automated Chart Generation**: Generate time-series charts for each performance metric
|
||||
- **Multi-Test Support**: Track performance for all test types (throughput, latency, accuracy)
|
||||
- **CSV Export**: Export performance data in structured CSV format
|
||||
- **Trend Analysis**: Visualize performance trends with interactive charts
|
||||
- **Comprehensive Metrics**: Track output throughput, E2E latency, TTFT, accept length, and more
|
||||
- **Time-Based Sampling**: Intelligent sampling strategy to cover extended time periods (up to 30 days) with limited API calls
|
||||
|
||||
### Test Balance Analyzer (`ci_analyzer_balance.py`)
|
||||
- **Time Gap Analysis**: Identify GPU tests with large gaps between elapsed and estimated times
|
||||
- **CI Balancing**: Help optimize CI by identifying tests that need time adjustments
|
||||
- **Gap Tracking**: Track maximum time gaps for each test across multiple CI runs
|
||||
- **PR Test Focus**: Only analyzes GPU jobs from pr-test.yml workflow (excludes AMD and other workflows)
|
||||
- **Ranking System**: Sort tests by time gap severity to prioritize adjustments
|
||||
- **CSV Export**: Export analysis results in CSV format for easy review
|
||||
- **GitHub Integration**: Generate GitHub Actions summaries with recommendations
|
||||
|
||||
### Failures Analyzer (`ci_failures_analysis.py`)
|
||||
- **Consecutive Failure Tracking**: Identify jobs currently failing
|
||||
- **Runner Health Monitoring**: Track runner failure rates and identify problematic infrastructure
|
||||
- **Multi-Workflow Support**: Monitors PR Test (Nvidia), PR Test (AMD), and PR Test (Xeon) workflows
|
||||
- **Queue Time Tracking**: Monitor average and P90 queue times per runner type
|
||||
- **Alert System**: Automatic alerts for consecutive failures and runner problems
|
||||
- **Instance Tracking**: Monitor specific runner instances for targeted remediation
|
||||
- **Slack Notifications**: Send condensed alerts to Slack (top 3 jobs/runners by consecutive failures and failure rates)
|
||||
- **GitHub Integration**: Generate comprehensive summaries with actionable recommendations
|
||||
- **JSON Export**: Export detailed analysis data for further processing
|
||||
|
||||
### Common Features
|
||||
- **Automated Monitoring**: GitHub Actions workflow for continuous CI and performance monitoring
|
||||
2. **Slack poster** (`post_ci_failures_to_slack.py`): Sends a condensed summary from a failure-analysis JSON to Slack (invoked by the workflow when `SGLANG_DIFFUSION_SLACK_TOKEN` is set).
|
||||
|
||||
## Installation
|
||||
|
||||
### For CI Analyzer
|
||||
No additional dependencies required beyond Python standard library and `requests`:
|
||||
|
||||
```bash
|
||||
pip install requests
|
||||
pip install requests slack_sdk
|
||||
```
|
||||
|
||||
### For Performance Analyzer
|
||||
Additional dependencies required for chart generation:
|
||||
|
||||
```bash
|
||||
pip install requests matplotlib pandas
|
||||
```
|
||||
|
||||
### For Test Balance Analyzer
|
||||
No additional dependencies required beyond Python standard library and `requests`:
|
||||
|
||||
```bash
|
||||
pip install requests
|
||||
```
|
||||
(`slack_sdk` is only required for `post_ci_failures_to_slack.py`.)
|
||||
|
||||
## Usage
|
||||
|
||||
### CI Analyzer
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
```bash
|
||||
# Replace YOUR_GITHUB_TOKEN with your actual token from https://github.com/settings/tokens
|
||||
python ci_analyzer.py --token YOUR_GITHUB_TOKEN
|
||||
```
|
||||
|
||||
#### Advanced Usage
|
||||
|
||||
```bash
|
||||
# Analyze last 1000 runs
|
||||
python ci_analyzer.py --token YOUR_GITHUB_TOKEN --limit 1000
|
||||
|
||||
# Custom output file
|
||||
python ci_analyzer.py --token YOUR_GITHUB_TOKEN --limit 500 --output my_analysis.json
|
||||
```
|
||||
|
||||
### Performance Analyzer
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
```bash
|
||||
# Analyze performance trends from recent CI runs
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN
|
||||
```
|
||||
|
||||
#### Advanced Usage
|
||||
|
||||
```bash
|
||||
# Analyze last 1000 PR Test runs (auto-enables uniform sampling for ~30 days coverage)
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --limit 1000
|
||||
|
||||
# Custom output directory
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --limit 500 --output-dir my_performance_data
|
||||
|
||||
# Use sampling with 500 runs (will use sequential mode since < 500 threshold)
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --limit 500
|
||||
|
||||
# Get ALL performance data within a specific date range (recommended for historical analysis)
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --start-date 2024-12-01 --end-date 2024-12-31
|
||||
|
||||
# Get complete data for the last week
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --start-date $(date -d '7 days ago' +%Y-%m-%d) --end-date $(date +%Y-%m-%d)
|
||||
|
||||
# Upload results to GitHub repository for sharing
|
||||
python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --limit 1000 --upload-to-github
|
||||
```
|
||||
|
||||
### Test Balance Analyzer
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
```bash
|
||||
# Analyze PR Test GPU job time gaps from recent CI runs
|
||||
python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN
|
||||
```
|
||||
|
||||
#### Advanced Usage
|
||||
|
||||
```bash
|
||||
# Analyze last 1000 PR Test GPU CI runs for comprehensive test balance analysis
|
||||
python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN --limit 1000
|
||||
|
||||
# Custom output file
|
||||
python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN --limit 500 --output my_balance_analysis.json
|
||||
```
|
||||
|
||||
### Failures Analyzer
|
||||
|
||||
#### Quick Start
|
||||
|
||||
```bash
|
||||
# Set token as environment variable (recommended for security)
|
||||
export GITHUB_TOKEN="your_token_here"
|
||||
|
||||
# Quick test with recent runs
|
||||
python ci_failures_analysis.py --token $GITHUB_TOKEN --limit 50 --threshold 2
|
||||
|
||||
# Standard analysis (same as automated workflow)
|
||||
python ci_failures_analysis.py --token $GITHUB_TOKEN --limit 300 --threshold 2
|
||||
|
||||
# Deep analysis
|
||||
python ci_failures_analysis.py --token $GITHUB_TOKEN --limit 500 --threshold 3
|
||||
```
|
||||
|
||||
#### Monitored Workflows
|
||||
### Slack notifications
|
||||
|
||||
The Failures Analyzer monitors the following workflows:
|
||||
|
||||
- **PR Test** - Nvidia GPU tests (self-hosted runners: 1-gpu-runner, 4-gpu-h100-runner, etc.)
|
||||
- **PR Test (AMD)** - AMD GPU tests (AMD-specific runners)
|
||||
- **PR Test (Xeon)** - Intel Xeon CPU tests (Xeon-specific runners)
|
||||
|
||||
All three workflows are analyzed together, with runner statistics tracked separately by runner type.
|
||||
|
||||
#### Slack Notifications
|
||||
|
||||
The Failures Analyzer can send condensed alerts to Slack. See [SLACK_SETUP.md](SLACK_SETUP.md) for complete setup instructions.
|
||||
|
||||
**What gets sent:**
|
||||
- Top 3 jobs with consecutive failures
|
||||
- Top 3 runners with consecutive failures
|
||||
- Top 3 jobs with highest total failure rate
|
||||
- Top 3 runners with highest total failure rate
|
||||
- Queue time summary
|
||||
From the `scripts/ci_monitor` directory, after generating a report:
|
||||
|
||||
```bash
|
||||
# Send Slack notification from analysis JSON
|
||||
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
|
||||
python slack_notifier.py --json ci_failure_analysis.json
|
||||
export SGLANG_DIFFUSION_SLACK_TOKEN="xoxb-..."
|
||||
python post_ci_failures_to_slack.py --report-file ci_failure_analysis_YYYYMMDD_HHMMSS.json
|
||||
```
|
||||
|
||||
#### Understanding the Output
|
||||
## Token permissions
|
||||
|
||||
The script generates a **2-section report**:
|
||||
The GitHub token needs `repo` and `workflow` scopes to read CI run data; otherwise API calls may return 404.
|
||||
|
||||
**Section 1: Currently Broken Jobs (Active Consecutive Failures)**
|
||||
- Shows consecutive failure streaks
|
||||
- These need immediate attention
|
||||
|
||||
**Section 2: Runner Health Analysis**
|
||||
- Shows which runners have high failure rates
|
||||
- Includes queue time metrics (average and P90)
|
||||
- Helps identify infrastructure vs code issues
|
||||
|
||||
#### Alert Types
|
||||
|
||||
**Job Alerts (Consecutive Failures):**
|
||||
- Triggered when a job fails ≥ threshold times in a row
|
||||
- Example: threshold=2, job fails 3 times → ALERT
|
||||
|
||||
**Runner Alerts:**
|
||||
- **Runner Health**: Runner has >30% failure rate with ≥2 different jobs failing
|
||||
- **Runner Instance**: Specific instance has >50% failure rate with ≥3 jobs
|
||||
|
||||
#### Output Files
|
||||
|
||||
- **Console**: Human-readable 3-section report (always generated)
|
||||
- **JSON**: Detailed data (optional, only if `--output` is specified)
|
||||
- **GitHub Summary**: Markdown (automatically generated in GitHub Actions)
|
||||
|
||||
**Important**: Make sure your GitHub token has `repo` and `workflow` permissions, otherwise you'll get 404 errors.
|
||||
|
||||
## Data Collection Strategies
|
||||
|
||||
The Performance Analyzer offers multiple strategies for collecting performance data to suit different analysis needs.
|
||||
|
||||
### 1. Uniform Sampling Strategy
|
||||
|
||||
**When to use**: Daily monitoring and trend analysis over extended periods.
|
||||
|
||||
- **Automatically enabled** when `--limit >= 500`
|
||||
- **Disabled** for smaller limits (< 500) to maintain backward compatibility
|
||||
|
||||
#### How it works:
|
||||
- Collects data uniformly across a 30-day period
|
||||
- Ensures even time distribution of samples
|
||||
- Provides consistent coverage for trend analysis
|
||||
|
||||
#### Example with 1000 Runs:
|
||||
- **Time Range**: Last 30 days
|
||||
- **Distribution**: 1000 samples evenly distributed across the period
|
||||
- **Coverage**: ~33 samples per day on average
|
||||
|
||||
### 2. Date Range Collection
|
||||
|
||||
**When to use**: Historical analysis, specific period investigation, or complete data collection.
|
||||
|
||||
Use `--start-date` and `--end-date` parameters to get **ALL** CI runs within a specific time range.
|
||||
|
||||
#### Features:
|
||||
- **Complete Data**: Gets every CI run in the specified range (no sampling)
|
||||
- **No Limit**: Ignores the `--limit` parameter
|
||||
- **Flexible Range**: Specify any date range you need
|
||||
- **Historical Analysis**: Perfect for investigating specific time periods
|
||||
|
||||
#### Date Format:
|
||||
- Use `YYYY-MM-DD` format (e.g., `2024-12-01`)
|
||||
- Both parameters are optional:
|
||||
- Only `--start-date`: Gets all runs from that date to now
|
||||
- Only `--end-date`: Gets all runs from 30 days ago to that date
|
||||
- Both: Gets all runs in the specified range
|
||||
|
||||
### 3. Sequential Collection (Traditional)
|
||||
|
||||
**When to use**: Quick checks or when you only need recent data.
|
||||
|
||||
- **Default behavior** for `--limit < 500`
|
||||
- Gets the most recent CI runs in chronological order
|
||||
- Fast and simple for immediate analysis
|
||||
|
||||
### Comparison
|
||||
|
||||
| Strategy | Use Case | Time Coverage | Data Completeness | API Efficiency |
|
||||
|----------|----------|---------------|-------------------|----------------|
|
||||
| **Uniform Sampling** | Daily monitoring, trends | ~30 days | Sampled | High |
|
||||
| **Date Range** | Historical analysis | Any range | Complete | Variable |
|
||||
| **Sequential** | Quick checks | 3-4 days | Complete (recent) | High |
|
||||
|
||||
### Benefits
|
||||
|
||||
- **Flexible Analysis**: Choose the right strategy for your needs
|
||||
- **Extended Coverage**: Up to 30 days with sampling, unlimited with date ranges
|
||||
- **Complete Data**: Get every run in a specific period when needed
|
||||
- **API Efficiency**: Optimized for different use patterns
|
||||
|
||||
## Parameters
|
||||
|
||||
### CI Analyzer Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--token` | Required | GitHub Personal Access Token |
|
||||
| `--limit` | 100 | Number of CI runs to analyze |
|
||||
| `--output` | ci_analysis.json | Output JSON file for detailed data |
|
||||
|
||||
### Performance Analyzer Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--token` | Required | GitHub Personal Access Token |
|
||||
| `--limit` | 100 | Number of PR Test runs to analyze (ignored when using date range) |
|
||||
| `--output-dir` | performance_tables | Output directory for CSV tables and PNG charts |
|
||||
| `--start-date` | None | Start date for date range query (YYYY-MM-DD format) |
|
||||
| `--end-date` | None | End date for date range query (YYYY-MM-DD format) |
|
||||
| `--upload-to-github` | False | Upload results to sglang-bot/sglang-ci-data repository |
|
||||
|
||||
### Test Balance Analyzer Parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--token` | Required | GitHub Personal Access Token |
|
||||
| `--limit` | 1000 | Number of CI runs to analyze |
|
||||
| `--output` | test_balance_report.json | Output JSON file for detailed analysis data |
|
||||
|
||||
### Failures Analyzer Parameters
|
||||
### Failures Analyzer parameters
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--token` | Required | GitHub Personal Access Token |
|
||||
| `--limit` | 500 | Number of workflow runs to analyze |
|
||||
| `--threshold` | 3 | Alert threshold for consecutive failures |
|
||||
| `--output` | None | Output JSON file (optional, only writes if specified) |
|
||||
| `--output` | None | Output JSON file (optional) |
|
||||
|
||||
## Getting GitHub Token
|
||||
## Historical note
|
||||
|
||||
1. Go to [GitHub Settings > Personal Access Tokens](https://github.com/settings/tokens)
|
||||
2. Click "Generate new token" > "Generate new token (classic)"
|
||||
3. **Important**: Select the following permissions:
|
||||
- `repo` (Full control of private repositories) - **Required for accessing repository data**
|
||||
- `workflow` (Update GitHub Action workflows) - **Required for reading CI/CD data**
|
||||
4. Copy the generated token and use it as `YOUR_GITHUB_TOKEN`
|
||||
|
||||
**Note**: Without the `repo` and `workflow` permissions, the tool will not be able to access CI run data and will return 404 errors.
|
||||
The former **CI Monitor** workflow (`ci-monitor.yml`) and its analyzers (`ci_analyzer.py`, `ci_analyzer_perf.py`, `ci_analyzer_balance.py`) were removed as redundant; use this failure monitor workflow and scripts for ongoing CI health alerts.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,534 +0,0 @@
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
import requests
|
||||
|
||||
|
||||
class SGLangTestBalanceAnalyzer:
|
||||
|
||||
def __init__(self, token: str):
|
||||
self.token = token
|
||||
self.base_url = "https://api.github.com"
|
||||
self.repo = "sgl-project/sglang"
|
||||
self.headers = {
|
||||
"Authorization": f"token {token}",
|
||||
"Accept": "application/vnd.github.v3+json",
|
||||
"User-Agent": "SGLang-Test-Balance-Analyzer/1.0",
|
||||
}
|
||||
self.session = requests.Session()
|
||||
self.session.headers.update(self.headers)
|
||||
|
||||
self.test_time_pattern = re.compile(
|
||||
r"filename='([^']+)',\s*elapsed=(\d+),\s*estimated_time=(\d+)"
|
||||
)
|
||||
|
||||
def get_recent_runs(self, limit: int = 1000) -> List[Dict]:
|
||||
print(f"Fetching {limit} recent CI runs...")
|
||||
|
||||
all_runs = []
|
||||
page = 1
|
||||
per_page = 100
|
||||
|
||||
while len(all_runs) < limit:
|
||||
url = f"{self.base_url}/repos/{self.repo}/actions/runs"
|
||||
params = {"per_page": min(per_page, limit - len(all_runs)), "page": page}
|
||||
|
||||
try:
|
||||
response = self.session.get(url, params=params)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
if not data.get("workflow_runs"):
|
||||
break
|
||||
|
||||
all_runs.extend(data["workflow_runs"])
|
||||
print(f"Fetched {len(all_runs)} runs so far...")
|
||||
|
||||
if len(data["workflow_runs"]) < per_page:
|
||||
break
|
||||
|
||||
page += 1
|
||||
time.sleep(0.1)
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"Error fetching CI data: {e}")
|
||||
break
|
||||
|
||||
return all_runs[:limit]
|
||||
|
||||
def get_job_logs(self, run_id: int, job_name: str) -> Optional[str]:
|
||||
try:
|
||||
jobs_url = f"{self.base_url}/repos/{self.repo}/actions/runs/{run_id}/jobs"
|
||||
response = self.session.get(jobs_url)
|
||||
response.raise_for_status()
|
||||
jobs_data = response.json()
|
||||
|
||||
target_job = None
|
||||
for job in jobs_data.get("jobs", []):
|
||||
if job.get("name", "") == job_name:
|
||||
target_job = job
|
||||
break
|
||||
|
||||
if not target_job:
|
||||
return None
|
||||
|
||||
logs_url = f"{self.base_url}/repos/{self.repo}/actions/jobs/{target_job['id']}/logs"
|
||||
response = self.session.get(logs_url)
|
||||
response.raise_for_status()
|
||||
|
||||
return response.text
|
||||
|
||||
except Exception as e:
|
||||
if "404" not in str(e):
|
||||
print(f"Failed to get job {job_name} logs: {e}")
|
||||
return None
|
||||
|
||||
def get_all_jobs_for_run(self, run_id: int) -> List[Dict]:
|
||||
try:
|
||||
jobs_url = f"{self.base_url}/repos/{self.repo}/actions/runs/{run_id}/jobs"
|
||||
response = self.session.get(jobs_url)
|
||||
response.raise_for_status()
|
||||
jobs_data = response.json()
|
||||
return jobs_data.get("jobs", [])
|
||||
except Exception as e:
|
||||
print(f"Failed to get jobs for run {run_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_job_logs_by_id(self, job_id: int) -> Optional[str]:
|
||||
try:
|
||||
logs_url = f"{self.base_url}/repos/{self.repo}/actions/jobs/{job_id}/logs"
|
||||
response = self.session.get(logs_url)
|
||||
response.raise_for_status()
|
||||
return response.text
|
||||
except Exception as e:
|
||||
if "404" not in str(e):
|
||||
print(f"Failed to get job {job_id} logs: {e}")
|
||||
return None
|
||||
|
||||
def parse_test_times(self, log_content: str) -> List[Dict]:
|
||||
if not log_content:
|
||||
return []
|
||||
|
||||
test_times = []
|
||||
matches = self.test_time_pattern.findall(log_content)
|
||||
filtered_count = 0
|
||||
|
||||
for match in matches:
|
||||
filename, elapsed_str, estimated_str = match
|
||||
try:
|
||||
elapsed = int(elapsed_str)
|
||||
estimated = int(estimated_str)
|
||||
gap = elapsed - estimated
|
||||
|
||||
if self._is_abnormal_test_data(
|
||||
elapsed, estimated, log_content, filename
|
||||
):
|
||||
filtered_count += 1
|
||||
continue
|
||||
|
||||
test_times.append(
|
||||
{
|
||||
"filename": filename,
|
||||
"elapsed": elapsed,
|
||||
"estimated": estimated,
|
||||
"gap": gap,
|
||||
}
|
||||
)
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
return test_times
|
||||
|
||||
def _is_abnormal_test_data(
|
||||
self, elapsed: int, estimated: int, log_content: str, filename: str
|
||||
) -> bool:
|
||||
|
||||
# To avoid collect retry data
|
||||
if elapsed % estimated == 0:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def collect_test_balance_data(self, runs: List[Dict]) -> Dict[str, Dict]:
|
||||
print("Starting test balance data collection...")
|
||||
|
||||
test_gaps = defaultdict(
|
||||
lambda: {
|
||||
"max_gap": 0,
|
||||
"max_elapsed": 0,
|
||||
"max_estimated": 0,
|
||||
"max_gap_run_info": {},
|
||||
"total_runs": 0,
|
||||
"all_gaps": [],
|
||||
}
|
||||
)
|
||||
|
||||
total_tests_parsed = 0
|
||||
abnormal_tests_filtered = 0
|
||||
|
||||
target_job_prefixes = [
|
||||
"stage-a-test-1",
|
||||
"unit-test-backend-1-gpu",
|
||||
"unit-test-backend-2-gpu",
|
||||
"stage-b-test-4-gpu-b200",
|
||||
"unit-test-backend-4-gpu",
|
||||
"unit-test-backend-8-gpu-h200",
|
||||
"unit-test-backend-8-gpu-h20",
|
||||
"unit-test-backend-4-gpu-b200",
|
||||
"unit-test-backend-4-gpu-gb200",
|
||||
"unit-test-deepep-4-gpu",
|
||||
"unit-test-deepep-8-gpu",
|
||||
"unit-test-backend-8-gpu-deepseek-v32",
|
||||
"performance-test-1-gpu-part-1",
|
||||
"performance-test-1-gpu-part-2",
|
||||
"performance-test-1-gpu-part-3",
|
||||
"performance-test-2-gpu",
|
||||
"accuracy-test-1-gpu",
|
||||
"accuracy-test-2-gpu",
|
||||
]
|
||||
|
||||
total_runs = len(runs)
|
||||
for i, run in enumerate(runs, 1):
|
||||
if i % 10 == 0 or i == total_runs:
|
||||
print(f"Processing run {i}/{total_runs}: #{run.get('run_number')}")
|
||||
|
||||
workflow_name = run.get("name", "")
|
||||
if "AMD" in workflow_name or "amd" in workflow_name.lower():
|
||||
continue
|
||||
|
||||
run_info = {
|
||||
"run_number": run.get("run_number"),
|
||||
"created_at": run.get("created_at"),
|
||||
"head_sha": run.get("head_sha", "")[:8],
|
||||
"author": run.get("head_commit", {})
|
||||
.get("author", {})
|
||||
.get("name", "Unknown"),
|
||||
"url": f"https://github.com/{self.repo}/actions/runs/{run.get('id')}",
|
||||
}
|
||||
|
||||
pull_requests = run.get("pull_requests", [])
|
||||
if pull_requests:
|
||||
run_info["pr_number"] = pull_requests[0].get("number")
|
||||
|
||||
all_jobs = self.get_all_jobs_for_run(run.get("id"))
|
||||
|
||||
for job in all_jobs:
|
||||
job_name = job.get("name", "")
|
||||
job_id = job.get("id")
|
||||
|
||||
matches_prefix = False
|
||||
for prefix in target_job_prefixes:
|
||||
if job_name.startswith(prefix):
|
||||
matches_prefix = True
|
||||
break
|
||||
|
||||
if not matches_prefix:
|
||||
continue
|
||||
|
||||
logs = self.get_job_logs_by_id(job_id)
|
||||
if not logs:
|
||||
continue
|
||||
|
||||
test_times = self.parse_test_times(logs)
|
||||
total_tests_parsed += len(test_times)
|
||||
|
||||
for test_data in test_times:
|
||||
filename = test_data["filename"]
|
||||
elapsed = test_data["elapsed"]
|
||||
estimated = test_data["estimated"]
|
||||
gap = test_data["gap"]
|
||||
|
||||
test_stats = test_gaps[filename]
|
||||
test_stats["total_runs"] += 1
|
||||
test_stats["all_gaps"].append(gap)
|
||||
|
||||
if gap > test_stats["max_gap"]:
|
||||
test_stats["max_gap"] = gap
|
||||
test_stats["max_elapsed"] = elapsed
|
||||
test_stats["max_estimated"] = estimated
|
||||
test_stats["max_gap_run_info"] = {
|
||||
**run_info,
|
||||
"job_name": job_name,
|
||||
"job_url": f"https://github.com/{self.repo}/actions/runs/{run.get('id')}/job/{job_id}",
|
||||
}
|
||||
|
||||
time.sleep(0.1)
|
||||
|
||||
return dict(test_gaps)
|
||||
|
||||
def generate_balance_report(
|
||||
self, test_data: Dict[str, Dict], output_file: str = "test_balance_report.json"
|
||||
):
|
||||
print("\n" + "=" * 80)
|
||||
print("SGLang Test Balance Analysis Report (PR Test GPU Jobs)")
|
||||
print("=" * 80)
|
||||
|
||||
sorted_tests = sorted(
|
||||
test_data.items(), key=lambda x: x[1]["max_gap"], reverse=True
|
||||
)
|
||||
|
||||
print(f"\nTotal tests analyzed: {len(sorted_tests)}")
|
||||
print(
|
||||
f"Tests with significant gaps (>100s): {len([t for t in sorted_tests if t[1]['max_gap'] > 100])}"
|
||||
)
|
||||
print(
|
||||
f"Tests with large gaps (>300s): {len([t for t in sorted_tests if t[1]['max_gap'] > 300])}"
|
||||
)
|
||||
print(
|
||||
f"Note: Abnormal test data (due to failures/retries) has been filtered out"
|
||||
)
|
||||
|
||||
report_data = {
|
||||
"summary": {
|
||||
"total_tests": len(sorted_tests),
|
||||
"tests_with_gaps_over_100s": len(
|
||||
[t for t in sorted_tests if t[1]["max_gap"] > 100]
|
||||
),
|
||||
"tests_with_gaps_over_300s": len(
|
||||
[t for t in sorted_tests if t[1]["max_gap"] > 300]
|
||||
),
|
||||
"analysis_timestamp": datetime.now().isoformat(),
|
||||
},
|
||||
"test_balance_table": [],
|
||||
}
|
||||
|
||||
print(f"\nTop 50 PR Test GPU Jobs with Largest Time Gaps:")
|
||||
print("-" * 100)
|
||||
print(
|
||||
f"{'Rank':<4} {'Test File':<40} {'Max Gap':<8} {'Max Elapsed':<12} {'Max Estimated':<15} {'Job Name':<25}"
|
||||
)
|
||||
print("-" * 100)
|
||||
|
||||
for i, (filename, stats) in enumerate(sorted_tests[:50], 1):
|
||||
test_name = filename.split("/")[-1] if "/" in filename else filename
|
||||
job_name = (
|
||||
stats["max_gap_run_info"].get("job_name", "Unknown")
|
||||
if stats["max_gap_run_info"]
|
||||
else "Unknown"
|
||||
)
|
||||
|
||||
print(
|
||||
f"{i:<4} {test_name:<40} {stats['max_gap']:<8} {stats['max_elapsed']:<12} {stats['max_estimated']:<15} {job_name:<25}"
|
||||
)
|
||||
|
||||
report_data["test_balance_table"].append(
|
||||
{
|
||||
"rank": i,
|
||||
"filename": filename,
|
||||
"test_name": test_name,
|
||||
"max_gap": stats["max_gap"],
|
||||
"max_elapsed": stats["max_elapsed"],
|
||||
"max_estimated": stats["max_estimated"],
|
||||
"max_gap_run_info": stats["max_gap_run_info"],
|
||||
"total_runs": stats["total_runs"],
|
||||
}
|
||||
)
|
||||
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
json.dump(report_data, f, ensure_ascii=False, indent=2)
|
||||
print(f"\nDetailed report saved to: {output_file}")
|
||||
|
||||
return report_data
|
||||
|
||||
def generate_github_summary(self, report_data: Dict):
|
||||
try:
|
||||
github_step_summary = os.environ.get("GITHUB_STEP_SUMMARY")
|
||||
if not github_step_summary:
|
||||
print("Not running in GitHub Actions, skipping summary generation")
|
||||
return
|
||||
|
||||
print("Generating GitHub Actions summary for Test Balance Analysis...")
|
||||
|
||||
summary_lines = []
|
||||
summary_lines.append(
|
||||
"# SGLang Test Balance Analysis Report (PR Test GPU Jobs)"
|
||||
)
|
||||
summary_lines.append("")
|
||||
summary_lines.append(
|
||||
f"**Analysis Timestamp:** {report_data['summary']['analysis_timestamp']}"
|
||||
)
|
||||
summary_lines.append("")
|
||||
|
||||
summary_lines.append("## Summary Statistics")
|
||||
summary_lines.append("")
|
||||
summary_lines.append("| Metric | Count |")
|
||||
summary_lines.append("|--------|-------|")
|
||||
summary_lines.append(
|
||||
f"| Total Tests Analyzed | {report_data['summary']['total_tests']} |"
|
||||
)
|
||||
summary_lines.append(
|
||||
f"| Tests with Gaps > 100s | {report_data['summary']['tests_with_gaps_over_100s']} |"
|
||||
)
|
||||
summary_lines.append(
|
||||
f"| Tests with Gaps > 300s | {report_data['summary']['tests_with_gaps_over_300s']} |"
|
||||
)
|
||||
summary_lines.append("")
|
||||
|
||||
summary_lines.append("## Top 30 PR Test GPU Jobs with Largest Time Gaps")
|
||||
summary_lines.append("")
|
||||
summary_lines.append(
|
||||
"| Rank | Test File | Max Gap (s) | Max Elapsed (s) | Max Estimated (s) | Job Name | Job Link | Total Runs |"
|
||||
)
|
||||
summary_lines.append(
|
||||
"|------|-----------|-------------|----------------|------------------|---------|----------|------------|"
|
||||
)
|
||||
|
||||
for test in report_data["test_balance_table"][:30]:
|
||||
test_name = test["test_name"]
|
||||
if len(test_name) > 30:
|
||||
test_name = test_name[:27] + "..."
|
||||
|
||||
job_name = (
|
||||
test["max_gap_run_info"].get("job_name", "Unknown")
|
||||
if test["max_gap_run_info"]
|
||||
else "Unknown"
|
||||
)
|
||||
job_url = (
|
||||
test["max_gap_run_info"].get("job_url", "")
|
||||
if test["max_gap_run_info"]
|
||||
else ""
|
||||
)
|
||||
job_link = f"[{job_name}]({job_url})" if job_url else job_name
|
||||
|
||||
summary_lines.append(
|
||||
f"| {test['rank']} | `{test_name}` | {test['max_gap']} | {test['max_elapsed']} | {test['max_estimated']} | {job_name} | [{job_name}]({job_url}) | {test['total_runs']} |"
|
||||
)
|
||||
|
||||
summary_lines.append("")
|
||||
summary_lines.append("## Recommendations")
|
||||
summary_lines.append("")
|
||||
summary_lines.append(
|
||||
"Based on the analysis above, consider adjusting estimated times for tests with large gaps:"
|
||||
)
|
||||
summary_lines.append("")
|
||||
|
||||
top_5_tests = report_data["test_balance_table"][:5]
|
||||
for test in top_5_tests:
|
||||
test_name = test["test_name"]
|
||||
if len(test_name) > 40:
|
||||
test_name = test_name[:37] + "..."
|
||||
suggested_estimated = test["max_elapsed"] + 50
|
||||
summary_lines.append(
|
||||
f"- **{test_name}**: Current max elapsed: {test['max_elapsed']}s, suggested estimated: {suggested_estimated}s"
|
||||
)
|
||||
|
||||
summary_lines.append("")
|
||||
summary_lines.append(
|
||||
"Set estimated times to be slightly higher than the maximum observed elapsed time to avoid CI timeouts."
|
||||
)
|
||||
|
||||
with open(github_step_summary, "w", encoding="utf-8") as f:
|
||||
f.write("\n".join(summary_lines))
|
||||
|
||||
print("GitHub Actions summary generated successfully")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Failed to generate GitHub Actions summary: {e}")
|
||||
|
||||
def save_csv_report(
|
||||
self, report_data: Dict, output_file: str = "test_balance_report.csv"
|
||||
):
|
||||
import csv
|
||||
|
||||
with open(output_file, "w", encoding="utf-8", newline="") as f:
|
||||
writer = csv.writer(f)
|
||||
|
||||
writer.writerow(
|
||||
[
|
||||
"Rank",
|
||||
"Test File",
|
||||
"Test Name",
|
||||
"Max Gap (s)",
|
||||
"Max Elapsed (s)",
|
||||
"Max Estimated (s)",
|
||||
"Job Name",
|
||||
"Max Gap Job URL",
|
||||
"Total Runs",
|
||||
]
|
||||
)
|
||||
|
||||
for test in report_data["test_balance_table"]:
|
||||
max_job_url = (
|
||||
test["max_gap_run_info"].get("job_url", "")
|
||||
if test["max_gap_run_info"]
|
||||
else ""
|
||||
)
|
||||
job_name = (
|
||||
test["max_gap_run_info"].get("job_name", "Unknown")
|
||||
if test["max_gap_run_info"]
|
||||
else "Unknown"
|
||||
)
|
||||
|
||||
writer.writerow(
|
||||
[
|
||||
test["rank"],
|
||||
test["filename"],
|
||||
test["test_name"],
|
||||
test["max_gap"],
|
||||
test["max_elapsed"],
|
||||
test["max_estimated"],
|
||||
job_name,
|
||||
max_job_url,
|
||||
test["total_runs"],
|
||||
]
|
||||
)
|
||||
|
||||
print(f"CSV report saved to: {output_file}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="SGLang Test Balance Analyzer")
|
||||
parser.add_argument("--token", required=True, help="GitHub Personal Access Token")
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=1000,
|
||||
help="Number of runs to analyze (default: 1000)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
default="test_balance_report.json",
|
||||
help="Output file (default: test_balance_report.json)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
analyzer = SGLangTestBalanceAnalyzer(args.token)
|
||||
|
||||
try:
|
||||
runs = analyzer.get_recent_runs(args.limit)
|
||||
|
||||
if not runs:
|
||||
print("No CI run data found")
|
||||
return
|
||||
|
||||
test_data = analyzer.collect_test_balance_data(runs)
|
||||
|
||||
if not test_data:
|
||||
print("No test balance data found")
|
||||
return
|
||||
|
||||
report_data = analyzer.generate_balance_report(test_data, args.output)
|
||||
|
||||
csv_output = args.output.replace(".json", ".csv")
|
||||
analyzer.save_csv_report(report_data, csv_output)
|
||||
|
||||
analyzer.generate_github_summary(report_data)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error during analysis: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -156,7 +156,7 @@ def post_ci_failures_to_slack(report_file: str) -> bool:
|
||||
if not hardware_jobs:
|
||||
summary = "✅ No critical failures detected in scheduled runs"
|
||||
if workflow_url:
|
||||
summary += f"\n<{workflow_url}|View CI Monitor Run>"
|
||||
summary += f"\n<{workflow_url}|View CI Failure Monitor run>"
|
||||
color = "good"
|
||||
else:
|
||||
# Ping relevant people when there are failures
|
||||
@@ -177,7 +177,9 @@ def post_ci_failures_to_slack(report_file: str) -> bool:
|
||||
summary_lines.append(f" • {test_type}: {job_list}")
|
||||
|
||||
if workflow_url:
|
||||
summary_lines.append(f"\n<{workflow_url}|View Full CI Monitor Report>")
|
||||
summary_lines.append(
|
||||
f"\n<{workflow_url}|View full CI Failure Monitor report>"
|
||||
)
|
||||
summary = "\n".join(summary_lines)
|
||||
color = "danger"
|
||||
|
||||
@@ -188,7 +190,7 @@ def post_ci_failures_to_slack(report_file: str) -> bool:
|
||||
attachments=[
|
||||
{
|
||||
"color": color,
|
||||
"footer": "SGLang CI Monitor",
|
||||
"footer": "SGLang CI Failure Monitor",
|
||||
"footer_icon": "https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png",
|
||||
"ts": int(datetime.now().timestamp()),
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user