resf-testing-repo/README.md

Sparrowdo Testing Orchestrator

Automated testing framework for Rocky Linux using Sparrowdo, QCOW2 images, and Jenkins.

Overview

This framework provides a production-ready continuous integration/testing system that:

• Automates VM Provisioning: Spins up isolated Rocky Linux VMs on-demand using QCOW2 images
• Runs Sparrowdo Tests in Parallel: Executes your test suite across multiple isolated environments simultaneously
• Supports Dynamic Configuration: Runtime customization of base images, preparation scripts, test selection, and concurrency
• Provides Web Interface: Jenkins UI for triggering builds, viewing results, and downloading logs
• Scales Across Machines: Multiple team members can connect their desktops as Jenkins agents

How It Works

User clicks "Build" in Jenkins
    ↓
Download QCOW2 base image
    ↓
Run custom prep script (install packages, configure services)
    ↓
Create golden image
    ↓
Create linked clones for each test (fast - copy-on-write)
    ↓
Run Sparrowdo tests in parallel (isolated VMs)
    ↓
Collect logs and archive results
    ↓
Auto-cleanup VMs and temporary images

Directory Structure

.
├── Jenkinsfile              # Jenkins Pipeline definition
├── README.md                # This file
├── scripts/
│   ├── setup_base.sh        # Prepares golden image from QCOW2
│   ├── provision_vm.sh      # Creates and starts test VM
│   └── cleanup_vm.sh        # Destroys VM and removes disk
├── tests/
│   └── (sample tests here)
└── docs/
    └── (additional documentation)

Prerequisites

On Jenkins Agent (Fedora/Rocky Linux Desktop)

1. KVM/QEMU/Libvirt

   sudo dnf install -y qemu-kvm libvirt virt-install libguestfs-tools-c
   sudo systemctl enable --now libvirtd
   sudo usermod -a -G libvirt $(whoami)
   

2. Sparrowdo

   # Install Raku (Perl 6)
   sudo dnf install -y rakudo
   
   # Install zef (Raku module manager)
   git clone https://github.com/ugexe/zef.git
   cd zef && raku -I. bin/zef install .
   
   # Install Sparrowdo
   zef install Sparrowdo
   

3. SSH Keys

   # Generate SSH key pair if needed
   ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
   

4. Jenkins Agent

   • Connect your Fedora desktop as a Jenkins agent
   • Label it as fedora-testing
   • Ensure the agent user has sudo access for virsh/libvirt commands

Quick Start

1. Clone Repository

cd ~/
git clone <your-repo-url> testing-orchestrator
cd testing-orchestrator

2. Test Scripts Locally (Optional)

# Make scripts executable
chmod +x scripts/*.sh

# Download a test QCOW2 image
cd /var/lib/libvirt/images
curl -LO https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2

# Create golden image
cd ~/testing-orchestrator
./scripts/setup_base.sh \
    /var/lib/libvirt/images/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 \
    "" \
    /var/lib/libvirt/images/golden-test.qcow2 \
    ~/.ssh/id_rsa.pub

# Provision a test VM
./scripts/provision_vm.sh test-vm-1 /var/lib/libvirt/images/golden-test.qcow2

# Clean up
./scripts/cleanup_vm.sh test-vm-1

3. Configure Jenkins Job

1. In Jenkins, create a new Pipeline job
2. Name it "rocky-sparrowdo-tests"
3. Under "Pipeline", select "Pipeline script from SCM"
4. Set SCM to "Git" and provide your repository URL
5. Set Script Path to "Jenkinsfile"
6. Save

4. Run Your First Build

1. Click "Build with Parameters"
2. Configure parameters:
   • QCOW2_URL: Rocky Linux base image URL
   • TEST_MATRIX: JSON array of your Sparrowdo test repositories
   • GOLDEN_PREP_SCRIPT: Customize image preparation
   • TEST_FILTER: Regex to filter which tests run
   • MAX_CONCURRENT: Number of parallel VMs
3. Click "Build"

Configuration

TEST_MATRIX Format

The TEST_MATRIX parameter accepts a JSON array of test definitions:

[
  {
    "name": "login-tests",
    "url": "https://github.com/your-org/login-tests.git",
    "branch": "main",
    "description": "SSH login and connectivity tests"
  },
  {
    "name": "database-tests",
    "url": "https://github.com/your-org/database-tests.git",
    "branch": "develop",
    "description": "PostgreSQL configuration tests"
  }
]

Each test repository should contain a sparrowfile at the root or in a subdirectory.

Custom Golden Image Preparation

The GOLDEN_PREP_SCRIPT parameter accepts a bash script that runs inside the QCOW2 image during preparation:

#!/bin/bash
set -e

# Update system
dnf update -y

# Install dependencies
dnf install -y perl git wget postgresql-server

# Configure services
systemctl enable postgresql
firewall-cmd --permanent --add-service=postgresql

echo "Custom preparation complete!"

Test Filtering

Use the TEST_FILTER parameter to run specific tests:

• .* - Run all tests
• login.* - Run all tests starting with "login"
• database-postgres - Run only the specific test
• (api|integration).* - Run API or integration tests

Features

Linked Clones for Speed

The framework uses QCOW2 linked clones (copy-on-write), which means:

• Creating a new VM takes seconds, not minutes
• Each test VM only stores differences from the golden image
• Disk space usage is minimal

Parallel Execution with Isolation

• Each test runs in its own isolated VM
• Tests cannot interfere with each other
• Concurrency is controlled via MAX_CONCURRENT parameter
• Jenkins lock mechanism prevents desktop overload

Automatic Cleanup

• VMs are automatically destroyed after each test
• Temporary images are removed after build
• Orphaned VMs from failed builds are cleaned up
• Optional: Keep golden image for debugging

Test Result Archival

• Each test's output is captured in logs/test.log
• Logs are archived as Jenkins artifacts
• Download logs directly from Jenkins UI

Troubleshooting

VM Won't Start

# Check libvirt status
sudo systemctl status libvirtd

# Verify image integrity
qemu-img info /var/lib/libvirt/images/golden-*.qcow2

# Check disk space
df -h /var/lib/libvirt/images/

VM Doesn't Get IP Address

# Check default network
sudo virsh net-list --all
sudo virsh net-start default  # If it's not running

# Verify DHCP
sudo virsh net-dhcp-leases default

SSH Connection Fails

# Test SSH manually (get IP from Jenkins logs)
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no root@<VM_IP>

# Check SSH key injection
sudo virt-cat -a /var/lib/libvirt/images/golden-*.qcow2 /root/.ssh/authorized_keys

Sparrowdo Test Fails

# View test logs in Jenkins artifacts
# Or run Sparrowdo manually against VM:
sparrowdo \
    --host=<VM_IP> \
    --ssh_user=root \
    --ssh_private_key=~/.ssh/id_rsa \
    --sparrowfile=/path/to/sparrowfile

Permission Denied Errors

# Ensure user is in libvirt group
sudo usermod -a -G libvirt $(whoami)
newgrp libvirt

# Fix image directory permissions
sudo chown -R $(whoami):$(whoami) /var/lib/libvirt/images

Jenkins Agent Offline

# Check agent service (if running as systemd service)
sudo systemctl status jenkins-agent

# View logs
sudo journalctl -u jenkins-agent -f

# Test connection from Jenkins controller
ssh jenkins@<agent-host> 'echo "Connection successful"'

Advanced Usage

Testing Beta Images

Change the QCOW2_URL parameter to point to beta images:

https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.4-beta.x86_64.qcow2

Custom VM Resources

Edit scripts/provision_vm.sh to adjust memory/CPU:

sudo virt-install \
    --name "$VM_NAME" \
    --memory 4096 \        # Increase memory
    --vcpus 4 \            # Increase CPUs
    ...

Preserving Golden Images for Debugging

1. Set KEEP_GOLDEN_IMAGE to true
2. After build, the golden image is preserved
3. Boot it manually for inspection:

VM_NAME="debug-vm"
sudo virt-install \
    --name "$VM_NAME" \
    --memory 2048 \
    --vcpus 2 \
    --disk /var/lib/libvirt/images/golden-<BUILD_ID>.qcow2 \
    --import \
    --os-variant rocky9-unknown \
    --network network=default

# Connect via console
sudo virsh console $VM_NAME

Running Tests Without Jenkins

# Manual test execution
./scripts/setup_base.sh \
    /path/to/base.qcow2 \
    /path/to/prep-script.sh \
    /path/to/golden.qcow2 \
    ~/.ssh/id_rsa.pub

IP=$(./scripts/provision_vm.sh my-test-vm /path/to/golden.qcow2)

sparrowdo \
    --host=$IP \
    --ssh_user=root \
    --ssh_private_key=~/.ssh/id_rsa \
    --sparrowfile=/path/to/test/sparrowfile

./scripts/cleanup_vm.sh my-test-vm

Contributing

Adding New Tests

1. Create a new Sparrowdo test repository
2. Add a sparrowfile at the root or in a subdirectory
3. Add the test to the TEST_MATRIX parameter in Jenkins

Modifying Scripts

1. Test changes locally first
2. Update documentation if behavior changes
3. Commit and push to repository
4. Jenkins will use updated scripts on next build

Support

For issues or questions:

• Check logs in Jenkins artifacts
• Review troubleshooting section above
• Verify prerequisites are installed correctly

License

[Specify your license here]

Authors

Rocky Linux Testing Team