resf-testing-repo/README.md

# Rocky Linux Testing Framework

Simple, portable Sparrowdo testing framework for Rocky Linux.

## Overview

This framework provides automated testing for Rocky Linux:

- **Simple Scripts**: 4 standalone bash scripts, easily portable
- **VM Isolation**: Each test runs in a fresh VM
- **Parallel Execution**: Run multiple tests concurrently
- **Fast Provisioning**: Linked clones (copy-on-write) for speed
- **Jenkins Ready**: Simple Jenkinsfile orchestrates scripts

## Prerequisites

```bash
sudo dnf install -y qemu-kvm libvirt virt-install guestfs-tools rakudo
sudo systemctl enable --now libvirtd
sudo usermod -a -G libvirt $(whoami)

ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
```

## Quick Start

### Manual Test Run

```bash
# 1. Download base image (cached automatically)
BASE=$(./scripts/download-image.sh \
  https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2)

# 2. Build golden image (includes Sparrowdo bootstrap)
./scripts/build-golden.sh "$BASE" /var/lib/libvirt/images/golden.qcow2

# 3. Run test
./scripts/run-test.sh my-test \
  https://github.com/your-org/test-repo.git \
  /var/lib/libvirt/images/golden.qcow2

# 4. Cleanup
./scripts/cleanup-all.sh
```

### Jenkins

1. Create new Pipeline job in Jenkins
2. Point to this repository
3. Use `Jenkinsfile.simple`
4. Configure parameters:
   - **QCOW2_URL**: Rocky Linux base image URL
   - **TEST_REPOS**: One test repo URL per line
   - **MAX_PARALLEL**: Number of concurrent tests
5. Build with Parameters

## How It Works

```
Download QCOW2 → Build Golden Image → Run Tests in Parallel → Cleanup
                 (includes bootstrap)   (isolated VMs)
```

**Golden Image Creation:**
1. Copy base QCOW2 image
2. Install Raku, Sparrowdo, dependencies via virt-customize
3. Create `rocky` user with sudo and SSH keys
4. Boot VM and run `sparrowdo --bootstrap` (once)
5. Shutdown VM, golden image ready

**Test Execution:**
1. Provision VM as linked clone (1 second)
2. Clone test repository
3. Run `sparrowdo --no_sudo` with test
4. Cleanup VM

## Scripts

### download-image.sh
```bash
./scripts/download-image.sh <url> [output_dir] [force]
```
Downloads and caches QCOW2 images. Returns path to cached image.

### build-golden.sh
```bash
./scripts/build-golden.sh <base_image> <golden_image> [ssh_pub_key]
```
Creates golden image from base QCOW2. Installs Raku/Sparrowdo and bootstraps.

### run-test.sh
```bash
./scripts/run-test.sh <test_name> <test_repo_url> <golden_image> [ssh_key]
```
Provisions VM, clones test repo, runs Sparrowdo test, cleans up.

### provision_vm.sh
```bash
./scripts/provision_vm.sh <vm_name> <golden_image> [timeout]
```
Creates VM as linked clone, starts it, returns IP address.

### cleanup_vm.sh
```bash
./scripts/cleanup_vm.sh <vm_name>
```
Destroys VM and removes disk image.

### cleanup-all.sh
```bash
./scripts/cleanup-all.sh [pattern]
```
Emergency cleanup for orphaned VMs matching pattern.

## Directory Structure

```
.
├── Jenkinsfile.simple       # Simple Jenkins pipeline
├── README.md                # This file
├── scripts/
│   ├── download-image.sh    # Download/cache images
│   ├── build-golden.sh      # Create golden image
│   ├── run-test.sh          # Run single test
│   ├── provision_vm.sh      # Provision VM
│   ├── cleanup_vm.sh        # Cleanup VM
│   └── cleanup-all.sh       # Emergency cleanup
└── docs/                    # Additional documentation
```

## Test Repository Format

Your test repository needs `sparrowfile` or `main.raku`:

```raku
#!/usr/bin/env raku
use Sparrowdo;

task-run 'check-sshd', %(
    plugin => 'systemd-service',
    args => ['sshd', 'running']
);

task-run 'verify-rocky-version', %(
    plugin => 'shell-command',
    args => 'cat /etc/rocky-release'
);
```

## Configuration

### Golden Image Users

- **rocky**: Primary test user (password: `rockypass`, sudo: yes)
- **root**: Root user (password: `rockytesting`)

Both have SSH keys injected from `~/.ssh/id_rsa.pub`.

### VM Resources

Default per VM (edit `provision_vm.sh` to change):
- Memory: 2048 MB
- CPUs: 2
- Disk: Linked clone (only diffs stored)

## Portability

All logic is in standalone shell scripts. Easy to integrate with:

**GitHub Actions:**
```yaml
- name: Run test
  run: ./scripts/run-test.sh my-test $REPO $GOLDEN
```

**GitLab CI:**
```yaml
test:
  script:
    - ./scripts/run-test.sh my-test $REPO $GOLDEN
```

**Windmill:**
```typescript
await bash.run(`./scripts/run-test.sh ${name} ${repo} ${golden}`)
```

**Manually:**
```bash
./scripts/run-test.sh my-test https://github.com/org/test.git golden.qcow2
```

## Troubleshooting

### List VMs
```bash
virsh -c qemu:///system list --all
```

### Get VM IP
```bash
virsh -c qemu:///system domifaddr <vm-name>
```

### SSH to VM
```bash
ssh -i ~/.ssh/id_rsa rocky@<vm-ip>
```

### View Console
```bash
virsh -c qemu:///system console <vm-name>
# Press Ctrl+] to exit
```

### Check Network
```bash
virsh -c qemu:///system net-list --all
virsh -c qemu:///system net-start default  # If stopped
```

### Force Cleanup
```bash
./scripts/cleanup-all.sh
rm -f /var/lib/libvirt/images/golden-*.qcow2
```

### Bootstrap Fails
```bash
# Check if VM is accessible
ssh -i ~/.ssh/id_rsa rocky@<vm-ip>

# Manually bootstrap
sparrowdo --host <vm-ip> --ssh_user rocky --bootstrap --color
```

## Performance

### Image Caching
- First download: ~5-10 minutes (2GB)
- Subsequent builds: 2 seconds (cached)

### Golden Image Build
- Base preparation: ~2-3 minutes (virt-customize)
- Bootstrap: ~5-10 minutes (sparrowdo --bootstrap)
- Total: ~7-13 minutes (once per build)

### Test Execution
- VM provision: ~30 seconds (boot + IP)
- Test runtime: Varies by test
- VM cleanup: ~2 seconds

### Example: 70 Tests
- Golden image: 10 minutes (once)
- 70 tests @ 3 concurrent: ~15 minutes
- **Total: 25 minutes**

Compare to bootstrapping each VM: 70 × 7 min = 490 minutes (8+ hours)!

## Advanced Usage

### Custom Golden Image
```bash
# Create custom prep script
cat > custom-prep.sh << 'EOF'
#!/bin/bash
dnf install -y postgresql-server nginx
systemctl enable postgresql nginx
EOF

# Use virt-customize directly
sudo virt-customize -a golden.qcow2 \
    --run custom-prep.sh \
    --selinux-relabel
```

### Testing Beta Images
```bash
BASE=$(./scripts/download-image.sh \
  https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base-9.5-beta.x86_64.qcow2 \
  /var/lib/libvirt/images true)  # Force re-download
```

### Running Subset of Tests
```bash
# In Jenkins, edit TEST_REPOS parameter
https://github.com/org/test1.git
https://github.com/org/test3.git
# (omit test2)
```

### Debugging Test Failures
```bash
# Run test manually and keep VM alive
VM_NAME="debug-vm-$$"
VM_IP=$(./scripts/provision_vm.sh "$VM_NAME" golden.qcow2 60 | tail -1)

# SSH to VM
ssh -i ~/.ssh/id_rsa rocky@$VM_IP

# Manually run test steps
cd /tmp
git clone https://github.com/org/test.git
cd test
sparrowdo --host localhost --ssh_user rocky --no_sudo --sparrowfile main.raku

# Cleanup when done
./scripts/cleanup_vm.sh "$VM_NAME"
```

## Contributing

This framework is designed to be simple and portable. Keep it that way:

- Scripts should be standalone bash
- Minimal dependencies
- Easy to read, no over-commenting
- Portable across CI/CD platforms

## License

[Your License Here]

## Authors

Rocky Linux Testing Team