resf-testing-repo/README.md

# Sparrowdo Testing Orchestrator

Automated testing framework for Rocky Linux using Sparrowdo, QCOW2 images, and Jenkins.

## Overview

This framework provides a production-ready continuous integration/testing system that:

- **Automates VM Provisioning**: Spins up isolated Rocky Linux VMs on-demand using QCOW2 images
- **Runs Sparrowdo Tests in Parallel**: Executes your test suite across multiple isolated environments simultaneously
- **Supports Dynamic Configuration**: Runtime customization of base images, preparation scripts, test selection, and concurrency
- **Provides Web Interface**: Jenkins UI for triggering builds, viewing results, and downloading logs
- **Scales Across Machines**: Multiple team members can connect their desktops as Jenkins agents

## How It Works

```
User clicks "Build" in Jenkins
    ↓
Download QCOW2 base image
    ↓
Run custom prep script (install packages, configure services)
    ↓
Create golden image
    ↓
Create linked clones for each test (fast - copy-on-write)
    ↓
Run Sparrowdo tests in parallel (isolated VMs)
    ↓
Collect logs and archive results
    ↓
Auto-cleanup VMs and temporary images
```

## Directory Structure

```
.
├── Jenkinsfile              # Jenkins Pipeline definition
├── README.md                # This file
├── scripts/
│   ├── setup_base.sh        # Prepares golden image from QCOW2
│   ├── provision_vm.sh      # Creates and starts test VM
│   └── cleanup_vm.sh        # Destroys VM and removes disk
├── tests/
│   └── (sample tests here)
└── docs/
    └── (additional documentation)
```

## Prerequisites

### On Jenkins Agent (Fedora/Rocky Linux Desktop)

1. **KVM/QEMU/Libvirt**
   ```bash
   sudo dnf install -y qemu-kvm libvirt virt-install libguestfs-tools-c
   sudo systemctl enable --now libvirtd
   sudo usermod -a -G libvirt $(whoami)
   ```

2. **Sparrowdo**
   ```bash
   # Install Raku (Perl 6)
   sudo dnf install -y rakudo

   # Install zef (Raku module manager)
   git clone https://github.com/ugexe/zef.git
   cd zef && raku -I. bin/zef install .

   # Install Sparrowdo
   zef install Sparrowdo
   ```

3. **SSH Keys**
   ```bash
   # Generate SSH key pair if needed
   ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
   ```

4. **Jenkins Agent**
   - Connect your Fedora desktop as a Jenkins agent
   - Label it as `fedora-testing`
   - Ensure the agent user has sudo access for virsh/libvirt commands

## Quick Start

### 1. Clone Repository

```bash
cd ~/
git clone <your-repo-url> testing-orchestrator
cd testing-orchestrator
```

### 2. Test Scripts Locally (Optional)

```bash
# Make scripts executable
chmod +x scripts/*.sh

# Download a test QCOW2 image
cd /var/lib/libvirt/images
curl -LO https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2

# Create golden image
cd ~/testing-orchestrator
./scripts/setup_base.sh \
    /var/lib/libvirt/images/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2 \
    "" \
    /var/lib/libvirt/images/golden-test.qcow2 \
    ~/.ssh/id_rsa.pub

# Provision a test VM
./scripts/provision_vm.sh test-vm-1 /var/lib/libvirt/images/golden-test.qcow2

# Clean up
./scripts/cleanup_vm.sh test-vm-1
```

### 3. Configure Jenkins Job

1. In Jenkins, create a new **Pipeline** job
2. Name it "rocky-sparrowdo-tests"
3. Under "Pipeline", select "Pipeline script from SCM"
4. Set SCM to "Git" and provide your repository URL
5. Set Script Path to "Jenkinsfile"
6. Save

### 4. Run Your First Build

1. Click "Build with Parameters"
2. Configure parameters:
   - **QCOW2_URL**: Rocky Linux base image URL
   - **TEST_MATRIX**: JSON array of your Sparrowdo test repositories
   - **GOLDEN_PREP_SCRIPT**: Customize image preparation
   - **TEST_FILTER**: Regex to filter which tests run
   - **MAX_CONCURRENT**: Number of parallel VMs
3. Click "Build"

## Configuration

### TEST_MATRIX Format

The `TEST_MATRIX` parameter accepts a JSON array of test definitions:

```json
[
  {
    "name": "login-tests",
    "url": "https://github.com/your-org/login-tests.git",
    "branch": "main",
    "description": "SSH login and connectivity tests"
  },
  {
    "name": "database-tests",
    "url": "https://github.com/your-org/database-tests.git",
    "branch": "develop",
    "description": "PostgreSQL configuration tests"
  }
]
```

Each test repository should contain a `sparrowfile` at the root or in a subdirectory.

### Custom Golden Image Preparation

The `GOLDEN_PREP_SCRIPT` parameter accepts a bash script that runs inside the QCOW2 image during preparation:

```bash
#!/bin/bash
set -e

# Update system
dnf update -y

# Install dependencies
dnf install -y perl git wget postgresql-server

# Configure services
systemctl enable postgresql
firewall-cmd --permanent --add-service=postgresql

echo "Custom preparation complete!"
```

### Test Filtering

Use the `TEST_FILTER` parameter to run specific tests:

- `.*` - Run all tests
- `login.*` - Run all tests starting with "login"
- `database-postgres` - Run only the specific test
- `(api|integration).*` - Run API or integration tests

## Features

### Linked Clones for Speed

The framework uses QCOW2 linked clones (copy-on-write), which means:
- Creating a new VM takes seconds, not minutes
- Each test VM only stores differences from the golden image
- Disk space usage is minimal

### Parallel Execution with Isolation

- Each test runs in its own isolated VM
- Tests cannot interfere with each other
- Concurrency is controlled via `MAX_CONCURRENT` parameter
- Jenkins lock mechanism prevents desktop overload

### Automatic Cleanup

- VMs are automatically destroyed after each test
- Temporary images are removed after build
- Orphaned VMs from failed builds are cleaned up
- Optional: Keep golden image for debugging

### Test Result Archival

- Each test's output is captured in `logs/test.log`
- Logs are archived as Jenkins artifacts
- Download logs directly from Jenkins UI

## Troubleshooting

### VM Won't Start

```bash
# Check libvirt status
sudo systemctl status libvirtd

# Verify image integrity
qemu-img info /var/lib/libvirt/images/golden-*.qcow2

# Check disk space
df -h /var/lib/libvirt/images/
```

### VM Doesn't Get IP Address

```bash
# Check default network
sudo virsh net-list --all
sudo virsh net-start default  # If it's not running

# Verify DHCP
sudo virsh net-dhcp-leases default
```

### SSH Connection Fails

```bash
# Test SSH manually (get IP from Jenkins logs)
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no root@<VM_IP>

# Check SSH key injection
sudo virt-cat -a /var/lib/libvirt/images/golden-*.qcow2 /root/.ssh/authorized_keys
```

### Sparrowdo Test Fails

```bash
# View test logs in Jenkins artifacts
# Or run Sparrowdo manually against VM:
sparrowdo \
    --host=<VM_IP> \
    --ssh_user=root \
    --ssh_private_key=~/.ssh/id_rsa \
    --sparrowfile=/path/to/sparrowfile
```

### Permission Denied Errors

```bash
# Ensure user is in libvirt group
sudo usermod -a -G libvirt $(whoami)
newgrp libvirt

# Fix image directory permissions
sudo chown -R $(whoami):$(whoami) /var/lib/libvirt/images
```

### Jenkins Agent Offline

```bash
# Check agent service (if running as systemd service)
sudo systemctl status jenkins-agent

# View logs
sudo journalctl -u jenkins-agent -f

# Test connection from Jenkins controller
ssh jenkins@<agent-host> 'echo "Connection successful"'
```

## Advanced Usage

### Testing Beta Images

Change the `QCOW2_URL` parameter to point to beta images:

```
https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.4-beta.x86_64.qcow2
```

### Custom VM Resources

Edit `scripts/provision_vm.sh` to adjust memory/CPU:

```bash
sudo virt-install \
    --name "$VM_NAME" \
    --memory 4096 \        # Increase memory
    --vcpus 4 \            # Increase CPUs
    ...
```

### Preserving Golden Images for Debugging

1. Set `KEEP_GOLDEN_IMAGE` to `true`
2. After build, the golden image is preserved
3. Boot it manually for inspection:

```bash
VM_NAME="debug-vm"
sudo virt-install \
    --name "$VM_NAME" \
    --memory 2048 \
    --vcpus 2 \
    --disk /var/lib/libvirt/images/golden-<BUILD_ID>.qcow2 \
    --import \
    --os-variant rocky9-unknown \
    --network network=default

# Connect via console
sudo virsh console $VM_NAME
```

### Running Tests Without Jenkins

```bash
# Manual test execution
./scripts/setup_base.sh \
    /path/to/base.qcow2 \
    /path/to/prep-script.sh \
    /path/to/golden.qcow2 \
    ~/.ssh/id_rsa.pub

IP=$(./scripts/provision_vm.sh my-test-vm /path/to/golden.qcow2)

sparrowdo \
    --host=$IP \
    --ssh_user=root \
    --ssh_private_key=~/.ssh/id_rsa \
    --sparrowfile=/path/to/test/sparrowfile

./scripts/cleanup_vm.sh my-test-vm
```

## Contributing

### Adding New Tests

1. Create a new Sparrowdo test repository
2. Add a `sparrowfile` at the root or in a subdirectory
3. Add the test to the `TEST_MATRIX` parameter in Jenkins

### Modifying Scripts

1. Test changes locally first
2. Update documentation if behavior changes
3. Commit and push to repository
4. Jenkins will use updated scripts on next build

## Support

For issues or questions:
- Check logs in Jenkins artifacts
- Review troubleshooting section above
- Verify prerequisites are installed correctly

## License

[Specify your license here]

## Authors

Rocky Linux Testing Team