admin/resf-testing-repo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates

Browse Source
This commit is contained in:
Stephen Simpson
2025-11-26 08:15:00 -06:00
parent 3cbd4525a0
commit bb829c9b63
18 changed files with 2440 additions and 349 deletions
Download Patch File Download Diff File Expand all files Collapse all files

553
README.md
View File

@@ -1,388 +1,325 @@
# Sparrowdo Testing Orchestrator
# Rocky Linux Testing Framework
Automated testing framework for Rocky Linux using Sparrowdo, QCOW2 images, and Jenkins.
Simple, portable Sparrowdo testing framework for Rocky Linux.
## Overview
This framework provides a production-ready continuous integration/testing system that:
This framework provides automated testing for Rocky Linux:
- **Automates VM Provisioning**: Spins up isolated Rocky Linux VMs on-demand using QCOW2 images
- **Runs Sparrowdo Tests in Parallel**: Executes your test suite across multiple isolated environments simultaneously
- **Supports Dynamic Configuration**: Runtime customization of base images, preparation scripts, test selection, and concurrency
- **Provides Web Interface**: Jenkins UI for triggering builds, viewing results, and downloading logs
- **Scales Across Machines**: Multiple team members can connect their desktops as Jenkins agents
- **Simple Scripts**: 4 standalone bash scripts, easily portable
- **VM Isolation**: Each test runs in a fresh VM
- **Parallel Execution**: Run multiple tests concurrently
- **Fast Provisioning**: Linked clones (copy-on-write) for speed
- **Jenkins Ready**: Simple Jenkinsfile orchestrates scripts
## Prerequisites
```bash
sudo dnf install -y qemu-kvm libvirt virt-install guestfs-tools rakudo
sudo systemctl enable --now libvirtd
sudo usermod -a -G libvirt $(whoami)
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
```
## Quick Start
### Manual Test Run
```bash
# 1. Download base image (cached automatically)
BASE=$(./scripts/download-image.sh \
https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2)
# 2. Build golden image (includes Sparrowdo bootstrap)
./scripts/build-golden.sh "$BASE" /var/lib/libvirt/images/golden.qcow2
# 3. Run test
./scripts/run-test.sh my-test \
https://github.com/your-org/test-repo.git \
/var/lib/libvirt/images/golden.qcow2
# 4. Cleanup
./scripts/cleanup-all.sh
```
### Jenkins
1. Create new Pipeline job in Jenkins
2. Point to this repository
3. Use `Jenkinsfile.simple`
4. Configure parameters:
- **QCOW2_URL**: Rocky Linux base image URL
- **TEST_REPOS**: One test repo URL per line
- **MAX_PARALLEL**: Number of concurrent tests
5. Build with Parameters
## How It Works
```
User clicks "Build" in Jenkins
↓
Download QCOW2 base image
↓
Run custom prep script (install packages, configure services)
↓
Create golden image
↓
Create linked clones for each test (fast - copy-on-write)
↓
Run Sparrowdo tests in parallel (isolated VMs)
↓
Collect logs and archive results
↓
Auto-cleanup VMs and temporary images
Download QCOW2 → Build Golden Image → Run Tests in Parallel → Cleanup
(includes bootstrap) (isolated VMs)
```
**Golden Image Creation:**
1. Copy base QCOW2 image
2. Install Raku, Sparrowdo, dependencies via virt-customize
3. Create `rocky` user with sudo and SSH keys
4. Boot VM and run `sparrowdo --bootstrap` (once)
5. Shutdown VM, golden image ready
**Test Execution:**
1. Provision VM as linked clone (1 second)
2. Clone test repository
3. Run `sparrowdo --no_sudo` with test
4. Cleanup VM
## Scripts
### download-image.sh
```bash
./scripts/download-image.sh <url> [output_dir] [force]
```
Downloads and caches QCOW2 images. Returns path to cached image.
### build-golden.sh
```bash
./scripts/build-golden.sh <base_image> <golden_image> [ssh_pub_key]
```
Creates golden image from base QCOW2. Installs Raku/Sparrowdo and bootstraps.
### run-test.sh
```bash
./scripts/run-test.sh <test_name> <test_repo_url> <golden_image> [ssh_key]
```
Provisions VM, clones test repo, runs Sparrowdo test, cleans up.
### provision_vm.sh
```bash
./scripts/provision_vm.sh <vm_name> <golden_image> [timeout]
```
Creates VM as linked clone, starts it, returns IP address.
### cleanup_vm.sh
```bash
./scripts/cleanup_vm.sh <vm_name>
```
Destroys VM and removes disk image.
### cleanup-all.sh
```bash
./scripts/cleanup-all.sh [pattern]
```
Emergency cleanup for orphaned VMs matching pattern.
## Directory Structure
```
.
├── Jenkinsfile # Jenkins Pipeline definition
├── Jenkinsfile.simple # Simple Jenkins pipeline
├── README.md # This file
├── scripts/
│ ├── setup_base.sh # Prepares golden image from QCOW2
│ ├── provision_vm.sh # Creates and starts test VM
│ └── cleanup_vm.sh # Destroys VM and removes disk
├── tests/
│ └── (sample tests here)
└── docs/
└── (additional documentation)
│ ├── download-image.sh # Download/cache images
│ ├── build-golden.sh # Create golden image
│ ├── run-test.sh # Run single test
│ ├── provision_vm.sh # Provision VM
│ ├── cleanup_vm.sh # Cleanup VM
│ └── cleanup-all.sh # Emergency cleanup
└── docs/ # Additional documentation
```
## Prerequisites
## Test Repository Format
### On Jenkins Agent (Fedora/Rocky Linux Desktop)
Your test repository needs `sparrowfile` or `main.raku`:
1. **KVM/QEMU/Libvirt**
```bash
sudo dnf install -y qemu-kvm libvirt virt-install libguestfs-tools-c
sudo systemctl enable --now libvirtd
sudo usermod -a -G libvirt $(whoami)
```
```raku
#!/usr/bin/env raku
use Sparrowdo;
2. **Sparrowdo**
```bash
# Install Raku (Perl 6)
sudo dnf install -y rakudo
task-run 'check-sshd', %(
plugin => 'systemd-service',
args => ['sshd', 'running']
);
# Install zef (Raku module manager)
git clone https://github.com/ugexe/zef.git
cd zef && raku -I. bin/zef install .
# Install Sparrowdo
zef install Sparrowdo
```
3. **SSH Keys**
```bash
# Generate SSH key pair if needed
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
```
4. **Jenkins Agent**
- Connect your Fedora desktop as a Jenkins agent
- Label it as `fedora-testing`
- Ensure the agent user has sudo access for virsh/libvirt commands
## Quick Start
### 1. Clone Repository
```bash
cd ~/
git clone <your-repo-url> testing-orchestrator
cd testing-orchestrator
task-run 'verify-rocky-version', %(
plugin => 'shell-command',
args => 'cat /etc/rocky-release'
);
```
### 2. Test Scripts Locally (Optional)
```bash
# Make scripts executable
chmod +x scripts/*.sh
# Download a test QCOW2 image
cd /var/lib/libvirt/images
curl -LO https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2
# Create golden image
cd ~/testing-orchestrator
./scripts/setup_base.sh \
/var/lib/libvirt/images/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 \
"" \
/var/lib/libvirt/images/golden-test.qcow2 \
~/.ssh/id_rsa.pub
# Provision a test VM
./scripts/provision_vm.sh test-vm-1 /var/lib/libvirt/images/golden-test.qcow2
# Clean up
./scripts/cleanup_vm.sh test-vm-1
```
### 3. Configure Jenkins Job
1. In Jenkins, create a new **Pipeline** job
2. Name it "rocky-sparrowdo-tests"
3. Under "Pipeline", select "Pipeline script from SCM"
4. Set SCM to "Git" and provide your repository URL
5. Set Script Path to "Jenkinsfile"
6. Save
### 4. Run Your First Build
1. Click "Build with Parameters"
2. Configure parameters:
- **QCOW2_URL**: Rocky Linux base image URL
- **TEST_MATRIX**: JSON array of your Sparrowdo test repositories
- **GOLDEN_PREP_SCRIPT**: Customize image preparation
- **TEST_FILTER**: Regex to filter which tests run
- **MAX_CONCURRENT**: Number of parallel VMs
3. Click "Build"
## Configuration
### TEST_MATRIX Format
### Golden Image Users
The `TEST_MATRIX` parameter accepts a JSON array of test definitions:
- **rocky**: Primary test user (password: `rockypass`, sudo: yes)
- **root**: Root user (password: `rockytesting`)
```json
[
{
"name": "login-tests",
"url": "https://github.com/your-org/login-tests.git",
"branch": "main",
"description": "SSH login and connectivity tests"
},
{
"name": "database-tests",
"url": "https://github.com/your-org/database-tests.git",
"branch": "develop",
"description": "PostgreSQL configuration tests"
}
]
Both have SSH keys injected from `~/.ssh/id_rsa.pub`.
### VM Resources
Default per VM (edit `provision_vm.sh` to change):
- Memory: 2048 MB
- CPUs: 2
- Disk: Linked clone (only diffs stored)
## Portability
All logic is in standalone shell scripts. Easy to integrate with:
**GitHub Actions:**
```yaml
- name: Run test
run: ./scripts/run-test.sh my-test $REPO $GOLDEN
```
Each test repository should contain a `sparrowfile` at the root or in a subdirectory.
**GitLab CI:**
```yaml
test:
script:
- ./scripts/run-test.sh my-test $REPO $GOLDEN
```
### Custom Golden Image Preparation
The `GOLDEN_PREP_SCRIPT` parameter accepts a bash script that runs inside the QCOW2 image during preparation:
**Windmill:**
```typescript
await bash.run(`./scripts/run-test.sh ${name} ${repo} ${golden}`)
```
**Manually:**
```bash
#!/bin/bash
set -e
# Update system
dnf update -y
# Install dependencies
dnf install -y perl git wget postgresql-server
# Configure services
systemctl enable postgresql
firewall-cmd --permanent --add-service=postgresql
echo "Custom preparation complete!"
./scripts/run-test.sh my-test https://github.com/org/test.git golden.qcow2
```
### Test Filtering
Use the `TEST_FILTER` parameter to run specific tests:
- `.*` - Run all tests
- `login.*` - Run all tests starting with "login"
- `database-postgres` - Run only the specific test
- `(api|integration).*` - Run API or integration tests
## Features
### Linked Clones for Speed
The framework uses QCOW2 linked clones (copy-on-write), which means:
- Creating a new VM takes seconds, not minutes
- Each test VM only stores differences from the golden image
- Disk space usage is minimal
### Parallel Execution with Isolation
- Each test runs in its own isolated VM
- Tests cannot interfere with each other
- Concurrency is controlled via `MAX_CONCURRENT` parameter
- Jenkins lock mechanism prevents desktop overload
### Automatic Cleanup
- VMs are automatically destroyed after each test
- Temporary images are removed after build
- Orphaned VMs from failed builds are cleaned up
- Optional: Keep golden image for debugging
### Test Result Archival
- Each test's output is captured in `logs/test.log`
- Logs are archived as Jenkins artifacts
- Download logs directly from Jenkins UI
## Troubleshooting
### VM Won't Start
### List VMs
```bash
# Check libvirt status
sudo systemctl status libvirtd
# Verify image integrity
qemu-img info /var/lib/libvirt/images/golden-*.qcow2
# Check disk space
df -h /var/lib/libvirt/images/
virsh -c qemu:///system list --all
```
### VM Doesn't Get IP Address
### Get VM IP
```bash
# Check default network
sudo virsh net-list --all
sudo virsh net-start default # If it's not running
# Verify DHCP
sudo virsh net-dhcp-leases default
virsh -c qemu:///system domifaddr <vm-name>
```
### SSH Connection Fails
### SSH to VM
```bash
# Test SSH manually (get IP from Jenkins logs)
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no root@<VM_IP>
# Check SSH key injection
sudo virt-cat -a /var/lib/libvirt/images/golden-*.qcow2 /root/.ssh/authorized_keys
ssh -i ~/.ssh/id_rsa rocky@<vm-ip>
```
### Sparrowdo Test Fails
### View Console
```bash
# View test logs in Jenkins artifacts
# Or run Sparrowdo manually against VM:
sparrowdo \
--host=<VM_IP> \
--ssh_user=root \
--ssh_private_key=~/.ssh/id_rsa \
--sparrowfile=/path/to/sparrowfile
virsh -c qemu:///system console <vm-name>
# Press Ctrl+] to exit
```
### Permission Denied Errors
### Check Network
```bash
# Ensure user is in libvirt group
sudo usermod -a -G libvirt $(whoami)
newgrp libvirt
# Fix image directory permissions
sudo chown -R $(whoami):$(whoami) /var/lib/libvirt/images
virsh -c qemu:///system net-list --all
virsh -c qemu:///system net-start default # If stopped
```
### Jenkins Agent Offline
### Force Cleanup
```bash
# Check agent service (if running as systemd service)
sudo systemctl status jenkins-agent
# View logs
sudo journalctl -u jenkins-agent -f
# Test connection from Jenkins controller
ssh jenkins@<agent-host> 'echo "Connection successful"'
./scripts/cleanup-all.sh
rm -f /var/lib/libvirt/images/golden-*.qcow2
```
### Bootstrap Fails
```bash
# Check if VM is accessible
ssh -i ~/.ssh/id_rsa rocky@<vm-ip>
# Manually bootstrap
sparrowdo --host <vm-ip> --ssh_user rocky --bootstrap --color
```
## Performance
### Image Caching
- First download: ~5-10 minutes (2GB)
- Subsequent builds: 2 seconds (cached)
### Golden Image Build
- Base preparation: ~2-3 minutes (virt-customize)
- Bootstrap: ~5-10 minutes (sparrowdo --bootstrap)
- Total: ~7-13 minutes (once per build)
### Test Execution
- VM provision: ~30 seconds (boot + IP)
- Test runtime: Varies by test
- VM cleanup: ~2 seconds
### Example: 70 Tests
- Golden image: 10 minutes (once)
- 70 tests @ 3 concurrent: ~15 minutes
- **Total: 25 minutes**
Compare to bootstrapping each VM: 70 × 7 min = 490 minutes (8+ hours)!
## Advanced Usage
### Custom Golden Image
```bash
# Create custom prep script
cat > custom-prep.sh << 'EOF'
#!/bin/bash
dnf install -y postgresql-server nginx
systemctl enable postgresql nginx
EOF
# Use virt-customize directly
sudo virt-customize -a golden.qcow2 \
--run custom-prep.sh \
--selinux-relabel
```
### Testing Beta Images
Change the `QCOW2_URL` parameter to point to beta images:
```
https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.4-beta.x86_64.qcow2
```
### Custom VM Resources
Edit `scripts/provision_vm.sh` to adjust memory/CPU:
```bash
sudo virt-install \
--name "$VM_NAME" \
--memory 4096 \ # Increase memory
--vcpus 4 \ # Increase CPUs
...
BASE=$(./scripts/download-image.sh \
https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.5-beta.x86_64.qcow2 \
/var/lib/libvirt/images true) # Force re-download
```
### Preserving Golden Images for Debugging
1. Set `KEEP_GOLDEN_IMAGE` to `true`
2. After build, the golden image is preserved
3. Boot it manually for inspection:
### Running Subset of Tests
```bash
VM_NAME="debug-vm"
sudo virt-install \
--name "$VM_NAME" \
--memory 2048 \
--vcpus 2 \
--disk /var/lib/libvirt/images/golden-<BUILD_ID>.qcow2 \
--import \
--os-variant rocky9-unknown \
--network network=default
# Connect via console
sudo virsh console $VM_NAME
# In Jenkins, edit TEST_REPOS parameter
https://github.com/org/test1.git
https://github.com/org/test3.git
# (omit test2)
```
### Running Tests Without Jenkins
### Debugging Test Failures
```bash
# Manual test execution
./scripts/setup_base.sh \
/path/to/base.qcow2 \
/path/to/prep-script.sh \
/path/to/golden.qcow2 \
~/.ssh/id_rsa.pub
# Run test manually and keep VM alive
VM_NAME="debug-vm-$$"
VM_IP=$(./scripts/provision_vm.sh "$VM_NAME" golden.qcow2 60 | tail -1)
IP=$(./scripts/provision_vm.sh my-test-vm /path/to/golden.qcow2)
# SSH to VM
ssh -i ~/.ssh/id_rsa rocky@$VM_IP
sparrowdo \
--host=$IP \
--ssh_user=root \
--ssh_private_key=~/.ssh/id_rsa \
--sparrowfile=/path/to/test/sparrowfile
# Manually run test steps
cd /tmp
git clone https://github.com/org/test.git
cd test
sparrowdo --host localhost --ssh_user rocky --no_sudo --sparrowfile main.raku
./scripts/cleanup_vm.sh my-test-vm
# Cleanup when done
./scripts/cleanup_vm.sh "$VM_NAME"
```
## Contributing
### Adding New Tests
This framework is designed to be simple and portable. Keep it that way:
1. Create a new Sparrowdo test repository
2. Add a `sparrowfile` at the root or in a subdirectory
3. Add the test to the `TEST_MATRIX` parameter in Jenkins
### Modifying Scripts
1. Test changes locally first
2. Update documentation if behavior changes
3. Commit and push to repository
4. Jenkins will use updated scripts on next build
## Support
For issues or questions:
- Check logs in Jenkins artifacts
- Review troubleshooting section above
- Verify prerequisites are installed correctly
- Scripts should be standalone bash
- Minimal dependencies
- Easy to read, no over-commenting
- Portable across CI/CD platforms
## License
[Specify your license here]
[Your License Here]
## Authors