Initial commit: Sparrowdo Testing Orchestrator framework
- Add VM provisioning, cleanup, and setup scripts - Add Jenkins pipeline with parameterized builds - Add comprehensive documentation - Support for parallel test execution with QCOW2 linked clones
This commit is contained in:
Stephen Simpson
389
README.md
Normal file
389
README.md
Normal file
@@ -0,0 +1,389 @@
|
||||
# Sparrowdo Testing Orchestrator
|
||||
|
||||
Automated testing framework for Rocky Linux using Sparrowdo, QCOW2 images, and Jenkins.
|
||||
|
||||
## Overview
|
||||
|
||||
This framework provides a production-ready continuous integration/testing system that:
|
||||
|
||||
- **Automates VM Provisioning**: Spins up isolated Rocky Linux VMs on-demand using QCOW2 images
|
||||
- **Runs Sparrowdo Tests in Parallel**: Executes your test suite across multiple isolated environments simultaneously
|
||||
- **Supports Dynamic Configuration**: Runtime customization of base images, preparation scripts, test selection, and concurrency
|
||||
- **Provides Web Interface**: Jenkins UI for triggering builds, viewing results, and downloading logs
|
||||
- **Scales Across Machines**: Multiple team members can connect their desktops as Jenkins agents
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
User clicks "Build" in Jenkins
|
||||
↓
|
||||
Download QCOW2 base image
|
||||
↓
|
||||
Run custom prep script (install packages, configure services)
|
||||
↓
|
||||
Create golden image
|
||||
↓
|
||||
Create linked clones for each test (fast - copy-on-write)
|
||||
↓
|
||||
Run Sparrowdo tests in parallel (isolated VMs)
|
||||
↓
|
||||
Collect logs and archive results
|
||||
↓
|
||||
Auto-cleanup VMs and temporary images
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── Jenkinsfile # Jenkins Pipeline definition
|
||||
├── README.md # This file
|
||||
├── scripts/
|
||||
│ ├── setup_base.sh # Prepares golden image from QCOW2
|
||||
│ ├── provision_vm.sh # Creates and starts test VM
|
||||
│ └── cleanup_vm.sh # Destroys VM and removes disk
|
||||
├── tests/
|
||||
│ └── (sample tests here)
|
||||
└── docs/
|
||||
└── (additional documentation)
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### On Jenkins Agent (Fedora/Rocky Linux Desktop)
|
||||
|
||||
1. **KVM/QEMU/Libvirt**
|
||||
```bash
|
||||
sudo dnf install -y qemu-kvm libvirt virt-install libguestfs-tools-c
|
||||
sudo systemctl enable --now libvirtd
|
||||
sudo usermod -a -G libvirt $(whoami)
|
||||
```
|
||||
|
||||
2. **Sparrowdo**
|
||||
```bash
|
||||
# Install Raku (Perl 6)
|
||||
sudo dnf install -y rakudo
|
||||
|
||||
# Install zef (Raku module manager)
|
||||
git clone https://github.com/ugexe/zef.git
|
||||
cd zef && raku -I. bin/zef install .
|
||||
|
||||
# Install Sparrowdo
|
||||
zef install Sparrowdo
|
||||
```
|
||||
|
||||
3. **SSH Keys**
|
||||
```bash
|
||||
# Generate SSH key pair if needed
|
||||
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
|
||||
```
|
||||
|
||||
4. **Jenkins Agent**
|
||||
- Connect your Fedora desktop as a Jenkins agent
|
||||
- Label it as `fedora-testing`
|
||||
- Ensure the agent user has sudo access for virsh/libvirt commands
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Clone Repository
|
||||
|
||||
```bash
|
||||
cd ~/
|
||||
git clone <your-repo-url> testing-orchestrator
|
||||
cd testing-orchestrator
|
||||
```
|
||||
|
||||
### 2. Test Scripts Locally (Optional)
|
||||
|
||||
```bash
|
||||
# Make scripts executable
|
||||
chmod +x scripts/*.sh
|
||||
|
||||
# Download a test QCOW2 image
|
||||
cd /var/lib/libvirt/images
|
||||
curl -LO https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2
|
||||
|
||||
# Create golden image
|
||||
cd ~/testing-orchestrator
|
||||
./scripts/setup_base.sh \
|
||||
/var/lib/libvirt/images/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 \
|
||||
"" \
|
||||
/var/lib/libvirt/images/golden-test.qcow2 \
|
||||
~/.ssh/id_rsa.pub
|
||||
|
||||
# Provision a test VM
|
||||
./scripts/provision_vm.sh test-vm-1 /var/lib/libvirt/images/golden-test.qcow2
|
||||
|
||||
# Clean up
|
||||
./scripts/cleanup_vm.sh test-vm-1
|
||||
```
|
||||
|
||||
### 3. Configure Jenkins Job
|
||||
|
||||
1. In Jenkins, create a new **Pipeline** job
|
||||
2. Name it "rocky-sparrowdo-tests"
|
||||
3. Under "Pipeline", select "Pipeline script from SCM"
|
||||
4. Set SCM to "Git" and provide your repository URL
|
||||
5. Set Script Path to "Jenkinsfile"
|
||||
6. Save
|
||||
|
||||
### 4. Run Your First Build
|
||||
|
||||
1. Click "Build with Parameters"
|
||||
2. Configure parameters:
|
||||
- **QCOW2_URL**: Rocky Linux base image URL
|
||||
- **TEST_MATRIX**: JSON array of your Sparrowdo test repositories
|
||||
- **GOLDEN_PREP_SCRIPT**: Customize image preparation
|
||||
- **TEST_FILTER**: Regex to filter which tests run
|
||||
- **MAX_CONCURRENT**: Number of parallel VMs
|
||||
3. Click "Build"
|
||||
|
||||
## Configuration
|
||||
|
||||
### TEST_MATRIX Format
|
||||
|
||||
The `TEST_MATRIX` parameter accepts a JSON array of test definitions:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "login-tests",
|
||||
"url": "https://github.com/your-org/login-tests.git",
|
||||
"branch": "main",
|
||||
"description": "SSH login and connectivity tests"
|
||||
},
|
||||
{
|
||||
"name": "database-tests",
|
||||
"url": "https://github.com/your-org/database-tests.git",
|
||||
"branch": "develop",
|
||||
"description": "PostgreSQL configuration tests"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Each test repository should contain a `sparrowfile` at the root or in a subdirectory.
|
||||
|
||||
### Custom Golden Image Preparation
|
||||
|
||||
The `GOLDEN_PREP_SCRIPT` parameter accepts a bash script that runs inside the QCOW2 image during preparation:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Update system
|
||||
dnf update -y
|
||||
|
||||
# Install dependencies
|
||||
dnf install -y perl git wget postgresql-server
|
||||
|
||||
# Configure services
|
||||
systemctl enable postgresql
|
||||
firewall-cmd --permanent --add-service=postgresql
|
||||
|
||||
echo "Custom preparation complete!"
|
||||
```
|
||||
|
||||
### Test Filtering
|
||||
|
||||
Use the `TEST_FILTER` parameter to run specific tests:
|
||||
|
||||
- `.*` - Run all tests
|
||||
- `login.*` - Run all tests starting with "login"
|
||||
- `database-postgres` - Run only the specific test
|
||||
- `(api|integration).*` - Run API or integration tests
|
||||
|
||||
## Features
|
||||
|
||||
### Linked Clones for Speed
|
||||
|
||||
The framework uses QCOW2 linked clones (copy-on-write), which means:
|
||||
- Creating a new VM takes seconds, not minutes
|
||||
- Each test VM only stores differences from the golden image
|
||||
- Disk space usage is minimal
|
||||
|
||||
### Parallel Execution with Isolation
|
||||
|
||||
- Each test runs in its own isolated VM
|
||||
- Tests cannot interfere with each other
|
||||
- Concurrency is controlled via `MAX_CONCURRENT` parameter
|
||||
- Jenkins lock mechanism prevents desktop overload
|
||||
|
||||
### Automatic Cleanup
|
||||
|
||||
- VMs are automatically destroyed after each test
|
||||
- Temporary images are removed after build
|
||||
- Orphaned VMs from failed builds are cleaned up
|
||||
- Optional: Keep golden image for debugging
|
||||
|
||||
### Test Result Archival
|
||||
|
||||
- Each test's output is captured in `logs/test.log`
|
||||
- Logs are archived as Jenkins artifacts
|
||||
- Download logs directly from Jenkins UI
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### VM Won't Start
|
||||
|
||||
```bash
|
||||
# Check libvirt status
|
||||
sudo systemctl status libvirtd
|
||||
|
||||
# Verify image integrity
|
||||
qemu-img info /var/lib/libvirt/images/golden-*.qcow2
|
||||
|
||||
# Check disk space
|
||||
df -h /var/lib/libvirt/images/
|
||||
```
|
||||
|
||||
### VM Doesn't Get IP Address
|
||||
|
||||
```bash
|
||||
# Check default network
|
||||
sudo virsh net-list --all
|
||||
sudo virsh net-start default # If it's not running
|
||||
|
||||
# Verify DHCP
|
||||
sudo virsh net-dhcp-leases default
|
||||
```
|
||||
|
||||
### SSH Connection Fails
|
||||
|
||||
```bash
|
||||
# Test SSH manually (get IP from Jenkins logs)
|
||||
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no root@<VM_IP>
|
||||
|
||||
# Check SSH key injection
|
||||
sudo virt-cat -a /var/lib/libvirt/images/golden-*.qcow2 /root/.ssh/authorized_keys
|
||||
```
|
||||
|
||||
### Sparrowdo Test Fails
|
||||
|
||||
```bash
|
||||
# View test logs in Jenkins artifacts
|
||||
# Or run Sparrowdo manually against VM:
|
||||
sparrowdo \
|
||||
--host=<VM_IP> \
|
||||
--ssh_user=root \
|
||||
--ssh_private_key=~/.ssh/id_rsa \
|
||||
--sparrowfile=/path/to/sparrowfile
|
||||
```
|
||||
|
||||
### Permission Denied Errors
|
||||
|
||||
```bash
|
||||
# Ensure user is in libvirt group
|
||||
sudo usermod -a -G libvirt $(whoami)
|
||||
newgrp libvirt
|
||||
|
||||
# Fix image directory permissions
|
||||
sudo chown -R $(whoami):$(whoami) /var/lib/libvirt/images
|
||||
```
|
||||
|
||||
### Jenkins Agent Offline
|
||||
|
||||
```bash
|
||||
# Check agent service (if running as systemd service)
|
||||
sudo systemctl status jenkins-agent
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u jenkins-agent -f
|
||||
|
||||
# Test connection from Jenkins controller
|
||||
ssh jenkins@<agent-host> 'echo "Connection successful"'
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Testing Beta Images
|
||||
|
||||
Change the `QCOW2_URL` parameter to point to beta images:
|
||||
|
||||
```
|
||||
https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.4-beta.x86_64.qcow2
|
||||
```
|
||||
|
||||
### Custom VM Resources
|
||||
|
||||
Edit `scripts/provision_vm.sh` to adjust memory/CPU:
|
||||
|
||||
```bash
|
||||
sudo virt-install \
|
||||
--name "$VM_NAME" \
|
||||
--memory 4096 \ # Increase memory
|
||||
--vcpus 4 \ # Increase CPUs
|
||||
...
|
||||
```
|
||||
|
||||
### Preserving Golden Images for Debugging
|
||||
|
||||
1. Set `KEEP_GOLDEN_IMAGE` to `true`
|
||||
2. After build, the golden image is preserved
|
||||
3. Boot it manually for inspection:
|
||||
|
||||
```bash
|
||||
VM_NAME="debug-vm"
|
||||
sudo virt-install \
|
||||
--name "$VM_NAME" \
|
||||
--memory 2048 \
|
||||
--vcpus 2 \
|
||||
--disk /var/lib/libvirt/images/golden-<BUILD_ID>.qcow2 \
|
||||
--import \
|
||||
--os-variant rocky9-unknown \
|
||||
--network network=default
|
||||
|
||||
# Connect via console
|
||||
sudo virsh console $VM_NAME
|
||||
```
|
||||
|
||||
### Running Tests Without Jenkins
|
||||
|
||||
```bash
|
||||
# Manual test execution
|
||||
./scripts/setup_base.sh \
|
||||
/path/to/base.qcow2 \
|
||||
/path/to/prep-script.sh \
|
||||
/path/to/golden.qcow2 \
|
||||
~/.ssh/id_rsa.pub
|
||||
|
||||
IP=$(./scripts/provision_vm.sh my-test-vm /path/to/golden.qcow2)
|
||||
|
||||
sparrowdo \
|
||||
--host=$IP \
|
||||
--ssh_user=root \
|
||||
--ssh_private_key=~/.ssh/id_rsa \
|
||||
--sparrowfile=/path/to/test/sparrowfile
|
||||
|
||||
./scripts/cleanup_vm.sh my-test-vm
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
### Adding New Tests
|
||||
|
||||
1. Create a new Sparrowdo test repository
|
||||
2. Add a `sparrowfile` at the root or in a subdirectory
|
||||
3. Add the test to the `TEST_MATRIX` parameter in Jenkins
|
||||
|
||||
### Modifying Scripts
|
||||
|
||||
1. Test changes locally first
|
||||
2. Update documentation if behavior changes
|
||||
3. Commit and push to repository
|
||||
4. Jenkins will use updated scripts on next build
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check logs in Jenkins artifacts
|
||||
- Review troubleshooting section above
|
||||
- Verify prerequisites are installed correctly
|
||||
|
||||
## License
|
||||
|
||||
[Specify your license here]
|
||||
|
||||
## Authors
|
||||
|
||||
Rocky Linux Testing Team
|
||||
Reference in New Issue
Block a user