Implement Ansible roles for Rocky Linux Testing Framework
- Added `bootstrap_sparrowdo` role for bootstrapping Sparrowdo on a VM. - Introduced `cleanup_vm` role for cleaning up VMs and disk images. - Created `download_image` role to download and cache QCOW2 images. - Developed `golden_image` role for creating and customizing golden images. - Implemented `provision_vm` role for provisioning VMs as linked clones. - Added `run_test` role for executing tests with Sparrowdo. - Created playbooks for building golden images, running single tests, and running test suites. - Enhanced documentation with usage examples, configuration details, and troubleshooting tips. - Added support for multiple cloud providers (AWS, Azure) in the test execution workflow. Signed-off-by: Stephen Simpson <ssimpson89@users.noreply.github.com>
This commit is contained in:
Stephen Simpson
562
docs/ANSIBLE-GUIDE.md
Normal file
562
docs/ANSIBLE-GUIDE.md
Normal file
@@ -0,0 +1,562 @@
|
||||
# Rocky Linux Testing Framework - Ansible Guide
|
||||
|
||||
This guide covers the Ansible-based implementation of the Rocky Linux Testing Framework.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Ansible Structure](#ansible-structure)
|
||||
- [Playbooks](#playbooks)
|
||||
- [Roles](#roles)
|
||||
- [Configuration](#configuration)
|
||||
- [Usage Examples](#usage-examples)
|
||||
- [Advanced Usage](#advanced-usage)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Overview
|
||||
|
||||
The Ansible implementation provides the same functionality as the shell scripts, but with:
|
||||
|
||||
- Better error handling and idempotency
|
||||
- Structured configuration management
|
||||
- Parallel test execution with better control
|
||||
- Reusable roles for composability
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
# Install required packages
|
||||
sudo dnf install -y ansible qemu-kvm libvirt virt-install guestfs-tools rakudo
|
||||
|
||||
# Enable libvirtd
|
||||
sudo systemctl enable --now libvirtd
|
||||
sudo usermod -a -G libvirt $(whoami)
|
||||
|
||||
# Generate SSH keys if not present
|
||||
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -N ""
|
||||
|
||||
# Install Ansible collections (if needed)
|
||||
ansible-galaxy collection install community.libvirt
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Build Golden Image
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
|
||||
# Build with defaults (Rocky 9)
|
||||
ansible-playbook playbooks/build-golden-image.yml
|
||||
|
||||
# Build with custom URL
|
||||
ansible-playbook playbooks/build-golden-image.yml \
|
||||
-e "qcow2_url=https://download.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud-Base.latest.x86_64.qcow2"
|
||||
```
|
||||
|
||||
### 2. Run Single Test
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/run-single-test.yml \
|
||||
-e "test_name=my-test" \
|
||||
-e "test_repo_url=https://github.com/your-org/test-repo.git"
|
||||
```
|
||||
|
||||
### 3. Run Test Suite
|
||||
|
||||
```bash
|
||||
# Edit playbooks/run-test-suite.yml to define your test matrix
|
||||
# Then run:
|
||||
ansible-playbook playbooks/run-test-suite.yml
|
||||
```
|
||||
|
||||
## Ansible Structure
|
||||
|
||||
```
|
||||
ansible/
|
||||
├── ansible.cfg # Ansible configuration
|
||||
├── inventory/
|
||||
│ ├── hosts.yml # Inventory file
|
||||
│ └── group_vars/
|
||||
│ └── all.yml # Global variables
|
||||
├── playbooks/
|
||||
│ ├── build-golden-image.yml # Build golden image workflow
|
||||
│ ├── run-single-test.yml # Run single test
|
||||
│ └── run-test-suite.yml # Run multiple tests
|
||||
└── roles/
|
||||
├── download_image/ # Download and cache QCOW2
|
||||
├── golden_image/ # Build golden image
|
||||
├── bootstrap_sparrowdo/ # Bootstrap Sparrowdo
|
||||
├── provision_vm/ # Provision test VM
|
||||
├── run_test/ # Run Sparrowdo test
|
||||
└── cleanup_vm/ # Cleanup VMs
|
||||
```
|
||||
|
||||
## Playbooks
|
||||
|
||||
### build-golden-image.yml
|
||||
|
||||
Complete workflow to create a golden image ready for testing.
|
||||
|
||||
**Variables:**
|
||||
- `qcow2_url`: URL of base QCOW2 image
|
||||
- `images_dir`: Directory for images (default: /var/lib/libvirt/images)
|
||||
- `golden_image_path`: Output path for golden image
|
||||
- `force_download`: Force re-download of base image
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml \
|
||||
-e "qcow2_url=https://..." \
|
||||
-e "force_download=true"
|
||||
```
|
||||
|
||||
### run-single-test.yml
|
||||
|
||||
Run a single Sparrowdo test against the golden image.
|
||||
|
||||
**Required Variables:**
|
||||
- `test_name`: Name of the test
|
||||
- `test_repo_url`: Git repository URL containing the test
|
||||
|
||||
**Optional Variables:**
|
||||
- `test_repo_branch`: Branch to use (default: main)
|
||||
- `golden_image_path`: Path to golden image
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
ansible-playbook playbooks/run-single-test.yml \
|
||||
-e "test_name=webserver-test" \
|
||||
-e "test_repo_url=https://github.com/org/webserver-test.git" \
|
||||
-e "test_repo_branch=develop"
|
||||
```
|
||||
|
||||
### run-test-suite.yml
|
||||
|
||||
Run multiple tests in parallel from a test matrix.
|
||||
|
||||
**Variables:**
|
||||
- `test_matrix`: List of tests to run
|
||||
- `max_parallel`: Maximum parallel tests
|
||||
- `test_filter`: Regex to filter tests
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
# Edit the playbook to define your tests:
|
||||
test_matrix:
|
||||
- name: "ssh-test"
|
||||
url: "https://github.com/org/ssh-test.git"
|
||||
branch: "main"
|
||||
description: "SSH service validation"
|
||||
- name: "network-test"
|
||||
url: "https://github.com/org/network-test.git"
|
||||
branch: "main"
|
||||
description: "Network configuration test"
|
||||
```
|
||||
|
||||
Then run:
|
||||
```bash
|
||||
ansible-playbook playbooks/run-test-suite.yml
|
||||
|
||||
# Or filter tests:
|
||||
ansible-playbook playbooks/run-test-suite.yml -e "test_filter=ssh.*"
|
||||
```
|
||||
|
||||
## Roles
|
||||
|
||||
### download_image
|
||||
|
||||
Downloads and caches QCOW2 images.
|
||||
|
||||
**Variables:**
|
||||
- `qcow2_url`: URL to download
|
||||
- `images_dir`: Cache directory
|
||||
- `force_download`: Force re-download
|
||||
- `image_path_var`: Variable name to store result
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
- include_role:
|
||||
name: download_image
|
||||
vars:
|
||||
qcow2_url: "https://..."
|
||||
image_path_var: "my_image_path"
|
||||
```
|
||||
|
||||
### golden_image
|
||||
|
||||
Creates a golden image using virt-customize.
|
||||
|
||||
**Variables:**
|
||||
- `base_image_path`: Source QCOW2 image (required)
|
||||
- `golden_image_path`: Output path
|
||||
- `ssh_public_key_path`: SSH key to inject
|
||||
- `custom_prep_script`: Custom preparation script
|
||||
- `use_default_prep`: Use default prep script
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
- include_role:
|
||||
name: golden_image
|
||||
vars:
|
||||
base_image_path: "/path/to/base.qcow2"
|
||||
golden_image_path: "/path/to/golden.qcow2"
|
||||
```
|
||||
|
||||
### bootstrap_sparrowdo
|
||||
|
||||
Bootstraps Sparrowdo on a golden image.
|
||||
|
||||
**Variables:**
|
||||
- `golden_image_path`: Golden image to bootstrap
|
||||
- `ssh_private_key_path`: SSH key for connection
|
||||
- `ssh_user`: User to connect as
|
||||
- `vm_boot_timeout`: Boot timeout in seconds
|
||||
- `bootstrap_timeout`: Bootstrap timeout
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
- include_role:
|
||||
name: bootstrap_sparrowdo
|
||||
vars:
|
||||
golden_image_path: "/path/to/golden.qcow2"
|
||||
```
|
||||
|
||||
### provision_vm
|
||||
|
||||
Provisions a VM as a linked clone.
|
||||
|
||||
**Variables:**
|
||||
- `vm_name`: VM name (required)
|
||||
- `golden_image_path`: Base image
|
||||
- `vm_memory`: Memory in MB
|
||||
- `vm_vcpus`: Number of vCPUs
|
||||
- `max_wait_ip`: Timeout for IP assignment
|
||||
- `vm_ip_var`: Variable name for returned IP
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
- include_role:
|
||||
name: provision_vm
|
||||
vars:
|
||||
vm_name: "test-vm-1"
|
||||
vm_ip_var: "test_vm_ip"
|
||||
|
||||
- debug:
|
||||
msg: "VM IP: {{ test_vm_ip }}"
|
||||
```
|
||||
|
||||
### run_test
|
||||
|
||||
Runs a complete test: provision -> test -> cleanup.
|
||||
|
||||
**Variables:**
|
||||
- `test_name`: Test name (required)
|
||||
- `test_repo_url`: Git repository (required)
|
||||
- `test_repo_branch`: Branch to use
|
||||
- `golden_image_path`: Golden image
|
||||
- `cleanup_after_test`: Cleanup VM after test
|
||||
- `save_logs`: Save test logs
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
- include_role:
|
||||
name: run_test
|
||||
vars:
|
||||
test_name: "my-test"
|
||||
test_repo_url: "https://github.com/org/test.git"
|
||||
```
|
||||
|
||||
### cleanup_vm
|
||||
|
||||
Cleans up VMs and disk images.
|
||||
|
||||
**Variables:**
|
||||
- `vm_name`: Single VM to cleanup
|
||||
- `cleanup_pattern`: Regex pattern for multiple VMs
|
||||
- `cleanup_vm_list`: List of VMs to cleanup
|
||||
- `force_destroy`: Force destroy running VMs
|
||||
- `remove_disk`: Remove disk images
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
# Cleanup single VM
|
||||
- include_role:
|
||||
name: cleanup_vm
|
||||
vars:
|
||||
vm_name: "test-vm-1"
|
||||
|
||||
# Cleanup by pattern
|
||||
- include_role:
|
||||
name: cleanup_vm
|
||||
vars:
|
||||
cleanup_pattern: "test-.*"
|
||||
|
||||
# Cleanup list
|
||||
- include_role:
|
||||
name: cleanup_vm
|
||||
vars:
|
||||
cleanup_vm_list:
|
||||
- "vm1"
|
||||
- "vm2"
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Inventory Variables
|
||||
|
||||
Edit `ansible/inventory/group_vars/all.yml`:
|
||||
|
||||
```yaml
|
||||
# VM defaults
|
||||
vm_memory: 4096 # Increase for heavy tests
|
||||
vm_vcpus: 4
|
||||
|
||||
# Parallel execution
|
||||
max_parallel: 5 # Run 5 tests concurrently
|
||||
|
||||
# Sparrowdo options
|
||||
sparrowdo_timeout: 1200 # Increase timeout
|
||||
sparrowdo_verbose: false # Less output
|
||||
```
|
||||
|
||||
### Per-Test Configuration
|
||||
|
||||
Override variables when running playbooks:
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/run-single-test.yml \
|
||||
-e "test_name=my-test" \
|
||||
-e "test_repo_url=https://..." \
|
||||
-e "vm_memory=4096" \
|
||||
-e "vm_vcpus=4" \
|
||||
-e "sparrowdo_timeout=1800"
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Example 1: Build Golden Image for Rocky 8
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml \
|
||||
-e "qcow2_url=https://download.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud-Base.latest.x86_64.qcow2" \
|
||||
-e "golden_image_path=/var/lib/libvirt/images/golden-rocky8.qcow2"
|
||||
```
|
||||
|
||||
### Example 2: Run Test with Custom Resources
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/run-single-test.yml \
|
||||
-e "test_name=heavy-test" \
|
||||
-e "test_repo_url=https://github.com/org/test.git" \
|
||||
-e "vm_memory=8192" \
|
||||
-e "vm_vcpus=8"
|
||||
```
|
||||
|
||||
### Example 3: Custom Test Suite
|
||||
|
||||
Create `my-test-suite.yml`:
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: My Custom Test Suite
|
||||
hosts: localhost
|
||||
connection: local
|
||||
gather_facts: true
|
||||
|
||||
vars:
|
||||
golden_image_path: "/var/lib/libvirt/images/golden.qcow2"
|
||||
|
||||
my_tests:
|
||||
- name: "database-test"
|
||||
url: "https://github.com/org/db-test.git"
|
||||
- name: "web-test"
|
||||
url: "https://github.com/org/web-test.git"
|
||||
|
||||
tasks:
|
||||
- name: Run each test
|
||||
include_role:
|
||||
name: run_test
|
||||
vars:
|
||||
test_name: "{{ item.name }}"
|
||||
test_repo_url: "{{ item.url }}"
|
||||
loop: "{{ my_tests }}"
|
||||
```
|
||||
|
||||
Run it:
|
||||
```bash
|
||||
ansible-playbook my-test-suite.yml
|
||||
```
|
||||
|
||||
### Example 4: Custom Golden Image Preparation
|
||||
|
||||
Create custom prep script `custom-prep.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
dnf install -y postgresql nginx redis
|
||||
systemctl enable postgresql nginx redis
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml \
|
||||
-e "custom_prep_script=/path/to/custom-prep.sh"
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Using Tags
|
||||
|
||||
Add tags to control execution:
|
||||
|
||||
```bash
|
||||
# Only download image
|
||||
ansible-playbook playbooks/build-golden-image.yml --tags download
|
||||
|
||||
# Skip bootstrap
|
||||
ansible-playbook playbooks/build-golden-image.yml --skip-tags bootstrap
|
||||
```
|
||||
|
||||
### Ansible Vault for Secrets
|
||||
|
||||
Store sensitive data in vault:
|
||||
|
||||
```bash
|
||||
ansible-vault create ansible/inventory/group_vars/secrets.yml
|
||||
```
|
||||
|
||||
Add passwords:
|
||||
```yaml
|
||||
root_password: "my-secure-password"
|
||||
rocky_user_password: "another-secure-password"
|
||||
```
|
||||
|
||||
Use it:
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml --ask-vault-pass
|
||||
```
|
||||
|
||||
### Cleanup Orphaned VMs
|
||||
|
||||
```bash
|
||||
ansible-playbook -m include_role -a name=cleanup_vm \
|
||||
-e "cleanup_pattern=test-.*" \
|
||||
localhost,
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check Ansible Syntax
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml --syntax-check
|
||||
```
|
||||
|
||||
### Dry Run (Check Mode)
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml --check
|
||||
```
|
||||
|
||||
### Verbose Output
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/run-single-test.yml -vvv \
|
||||
-e "test_name=my-test" \
|
||||
-e "test_repo_url=https://..."
|
||||
```
|
||||
|
||||
### List Tasks
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/build-golden-image.yml --list-tasks
|
||||
```
|
||||
|
||||
### Debugging Failed Tests
|
||||
|
||||
```bash
|
||||
# Keep VM after failure
|
||||
ansible-playbook playbooks/run-single-test.yml \
|
||||
-e "test_name=my-test" \
|
||||
-e "test_repo_url=https://..." \
|
||||
-e "cleanup_after_test=false"
|
||||
|
||||
# Then SSH to investigate
|
||||
ssh -i ~/.ssh/id_rsa rocky@<vm-ip>
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue: "Golden image not found"**
|
||||
```bash
|
||||
# Build golden image first
|
||||
ansible-playbook playbooks/build-golden-image.yml
|
||||
```
|
||||
|
||||
**Issue: "VM failed to get IP"**
|
||||
```bash
|
||||
# Check libvirt network
|
||||
sudo virsh net-list --all
|
||||
sudo virsh net-start default
|
||||
|
||||
# Increase timeout
|
||||
ansible-playbook ... -e "max_wait_ip=60"
|
||||
```
|
||||
|
||||
**Issue: "Bootstrap timeout"**
|
||||
```bash
|
||||
# Increase timeout
|
||||
ansible-playbook playbooks/build-golden-image.yml \
|
||||
-e "bootstrap_timeout=1800"
|
||||
```
|
||||
|
||||
**Issue: "Parallel tests failing"**
|
||||
```bash
|
||||
# Reduce parallelism
|
||||
ansible-playbook playbooks/run-test-suite.yml \
|
||||
-e "max_parallel=1"
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Cache base images**: First download is slow, subsequent runs are fast
|
||||
2. **Reuse golden images**: Build once, test many times
|
||||
3. **Tune parallel execution**: Balance between speed and resource usage
|
||||
4. **Use local mirrors**: Speed up package installation in prep scripts
|
||||
5. **Disable verbose logging**: For faster execution in production
|
||||
|
||||
## Migration from Shell Scripts
|
||||
|
||||
| Shell Script | Ansible Equivalent |
|
||||
|--------------|-------------------|
|
||||
| `download-image.sh` | Role: `download_image` |
|
||||
| `setup_base.sh` | Role: `golden_image` |
|
||||
| `bootstrap_golden.sh` | Role: `bootstrap_sparrowdo` |
|
||||
| `provision_vm.sh` | Role: `provision_vm` |
|
||||
| `run-test.sh` | Role: `run_test` |
|
||||
| `cleanup_vm.sh` | Role: `cleanup_vm` |
|
||||
| `cleanup-all.sh` | Role: `cleanup_vm` with pattern |
|
||||
| Manual workflow | Playbook: `build-golden-image.yml` |
|
||||
| Test execution | Playbooks: `run-single-test.yml`, `run-test-suite.yml` |
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new functionality:
|
||||
|
||||
1. Create a new role in `ansible/roles/`
|
||||
2. Add defaults in `defaults/main.yml`
|
||||
3. Document variables in role README
|
||||
4. Create example playbook in `playbooks/`
|
||||
5. Update this documentation
|
||||
|
||||
## License
|
||||
|
||||
[Your License Here]
|
||||
|
||||
## Authors
|
||||
|
||||
Rocky Linux Testing Team
|
||||
@@ -108,30 +108,21 @@ sparrowdo --host $VM_IP --ssh_user rocky --no_sudo --sparrowfile test.raku
|
||||
./scripts/cleanup_vm.sh test-vm-1
|
||||
```
|
||||
|
||||
## Jenkins Pipeline Flow
|
||||
## Workflow
|
||||
|
||||
The Jenkinsfile automatically handles bootstrap:
|
||||
Bootstrap is handled automatically in the build process:
|
||||
|
||||
```groovy
|
||||
stage('Prepare Golden Image') {
|
||||
// Creates golden image with Raku/zef
|
||||
setup_base.sh → golden.qcow2 (with Raku)
|
||||
}
|
||||
1. **Prepare Golden Image**
|
||||
- setup_base.sh → golden.qcow2 (with Raku)
|
||||
|
||||
stage('Bootstrap Golden Image') {
|
||||
// Bootstraps Sparrowdo ONCE
|
||||
bootstrap_golden.sh → golden.qcow2 (with Sparrowdo)
|
||||
}
|
||||
2. **Bootstrap Golden Image**
|
||||
- bootstrap_golden.sh → golden.qcow2 (with Sparrowdo)
|
||||
|
||||
stage('Run Tests') {
|
||||
parallel {
|
||||
test1: provision → run test → cleanup
|
||||
test2: provision → run test → cleanup
|
||||
test3: provision → run test → cleanup
|
||||
// No bootstrap in any test!
|
||||
}
|
||||
}
|
||||
```
|
||||
3. **Run Tests in Parallel**
|
||||
- provision → run test → cleanup
|
||||
- provision → run test → cleanup
|
||||
- provision → run test → cleanup
|
||||
- (No bootstrap in any test!)
|
||||
|
||||
## Time Savings Example
|
||||
|
||||
@@ -272,7 +263,7 @@ The bootstrap script outputs:
|
||||
|
||||
If any step fails, the golden image is NOT bootstrapped. Check logs and retry.
|
||||
|
||||
## Integration with CI/CD
|
||||
## Integration with Automation
|
||||
|
||||
### Nightly Golden Image Rebuild
|
||||
```bash
|
||||
@@ -287,10 +278,11 @@ If any step fails, the golden image is NOT bootstrapped. Check logs and retry.
|
||||
# Provisions temp VM, runs one test, destroys
|
||||
```
|
||||
|
||||
### Jenkins Scheduled Build
|
||||
```groovy
|
||||
// Rebuild golden images weekly
|
||||
cron('H 2 * * 0') // Sunday 2 AM
|
||||
### Scheduled Builds
|
||||
Use your automation tool (cron, systemd timers, etc.) to rebuild golden images periodically:
|
||||
```bash
|
||||
# Weekly golden image rebuild
|
||||
0 2 * * 0 /path/to/repo/ansible/playbooks/build-golden-image.yml
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
@@ -93,7 +93,7 @@ sparrowdo \
|
||||
### 3. Virsh Connection Issues ✅
|
||||
- **Problem**: Non-root users couldn't access libvirt without explicit URI
|
||||
- **Solution**: Added `-c qemu:///system` to all virsh commands
|
||||
- **Impact**: Scripts work for jenkins user and other non-root users
|
||||
- **Impact**: Scripts work for non-root users
|
||||
|
||||
### 4. Image Caching ✅
|
||||
- **Problem**: Framework re-downloaded QCOW2 images on every build
|
||||
@@ -109,17 +109,14 @@ sparrowdo \
|
||||
## File Changes
|
||||
|
||||
### Modified Files
|
||||
- `Jenkinsfile` - Added bootstrap step, rocky user, main.raku support
|
||||
- `scripts/setup_base.sh` - Inject SSH keys for rocky user
|
||||
- `scripts/provision_vm.sh` - Fixed set -e issue, improved error handling
|
||||
- `scripts/cleanup_vm.sh` - Added explicit qemu:///system connection
|
||||
- `docs/default-prep.sh` - Create rocky user, remove D-Bus commands
|
||||
- `docs/manual-test-run.sh` - Add bootstrap step, use rocky user
|
||||
- `README.md` - Updated prerequisites (guestfs-tools)
|
||||
|
||||
### New Files
|
||||
- `docs/virt-customize-guide.md` - Comprehensive guide on offline image customization
|
||||
- `docs/manual-steps.md` - Step-by-step manual testing guide
|
||||
- `docs/CHANGES.md` - This file
|
||||
|
||||
## Testing Checklist
|
||||
@@ -162,20 +159,11 @@ sparrowdo --host $VM_IP --ssh_user rocky --bootstrap --color
|
||||
### For Existing Tests
|
||||
If you have existing Sparrowdo tests that assumed root user:
|
||||
|
||||
1. **Update TEST_MATRIX** in Jenkins to use rocky user
|
||||
1. **Ensure tests use rocky user** - Update SSH commands to use rocky@host
|
||||
2. **Ensure tests use --no_sudo** flag
|
||||
3. **Add bootstrap step** before test execution
|
||||
4. **Verify sudoers access** if tests need elevated privileges
|
||||
|
||||
### Jenkins Pipeline Changes
|
||||
The Jenkinsfile automatically handles:
|
||||
- Creating rocky user in golden image
|
||||
- Injecting SSH keys for rocky user
|
||||
- Running bootstrap before tests
|
||||
- Using `--no_sudo` flag with rocky user
|
||||
|
||||
No manual intervention needed for Jenkins builds.
|
||||
|
||||
## Performance Improvements
|
||||
|
||||
### Image Caching
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "Preparing standard golden image..."
|
||||
|
||||
# NOTE: This script runs inside virt-customize (offline mode)
|
||||
# Cannot use systemctl, firewall-cmd, or other D-Bus dependent commands
|
||||
# Use systemctl enable only (works offline), or direct file manipulation
|
||||
|
||||
# Update system (optional - can be slow)
|
||||
# Uncomment if you want latest packages:
|
||||
# dnf update -y
|
||||
|
||||
# Install common testing dependencies including Raku/Sparrowdo
|
||||
dnf install -y \
|
||||
perl \
|
||||
git \
|
||||
wget \
|
||||
tar \
|
||||
openssh-server \
|
||||
vim \
|
||||
rakudo \
|
||||
rakudo-zef
|
||||
|
||||
# Enable services (these work in offline mode)
|
||||
# systemctl enable works by creating symlinks, no D-Bus needed
|
||||
systemctl enable sshd
|
||||
|
||||
# Create rocky user (standard non-root user for Rocky Linux)
|
||||
useradd -m rocky 2>/dev/null || true
|
||||
echo "rocky:rockypass" | chpasswd
|
||||
|
||||
# Add rocky user to sudoers
|
||||
echo "rocky ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/rocky
|
||||
chmod 0440 /etc/sudoers.d/rocky
|
||||
|
||||
# Install Sparrowdo for rocky user
|
||||
# Note: This needs to run as rocky user, but we're in offline mode
|
||||
# So we prepare the environment, and actual Sparrowdo install happens on first boot
|
||||
mkdir -p /home/rocky/.sparrowdo-bootstrap
|
||||
cat > /home/rocky/.sparrowdo-bootstrap/install.sh << 'BOOTSTRAP_EOF'
|
||||
#!/bin/bash
|
||||
# This script will be run on first boot by rocky user
|
||||
if [ ! -f /home/rocky/.sparrowdo-installed ]; then
|
||||
zef install --/test Sparrowdo
|
||||
touch /home/rocky/.sparrowdo-installed
|
||||
fi
|
||||
BOOTSTRAP_EOF
|
||||
chmod +x /home/rocky/.sparrowdo-bootstrap/install.sh
|
||||
chown -R rocky:rocky /home/rocky/.sparrowdo-bootstrap
|
||||
|
||||
# Create testuser for backward compatibility
|
||||
useradd -m testuser 2>/dev/null || true
|
||||
echo "testuser:testpass" | chpasswd
|
||||
|
||||
# Add testuser to sudoers
|
||||
echo "testuser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/testuser
|
||||
chmod 0440 /etc/sudoers.d/testuser
|
||||
|
||||
echo "Golden image preparation complete!"
|
||||
echo "NOTE: Sparrowdo will be installed on first SSH connection by rocky user"
|
||||
@@ -1,431 +0,0 @@
|
||||
# Manual Test Run - Step by Step
|
||||
|
||||
This guide provides the exact commands Jenkins runs, so you can execute them manually for testing.
|
||||
|
||||
## Quick Start - Automated Script
|
||||
|
||||
I've created an interactive script that walks through all steps:
|
||||
|
||||
```bash
|
||||
cd /Users/ssimpson/Documents/git/resf-test-jenkins
|
||||
|
||||
# Edit the configuration in the script first:
|
||||
vim docs/manual-test-run.sh
|
||||
# Update: TEST_REPO_URL, TEST_REPO_BRANCH, TEST_NAME
|
||||
|
||||
# Run the interactive script:
|
||||
./docs/manual-test-run.sh
|
||||
```
|
||||
|
||||
The script will pause at each step so you can see what's happening.
|
||||
|
||||
---
|
||||
|
||||
## Manual Step-by-Step Commands
|
||||
|
||||
If you prefer to run each command manually, here's the exact sequence:
|
||||
|
||||
### Prerequisites Check
|
||||
|
||||
```bash
|
||||
# Verify required tools
|
||||
which qemu-img virsh virt-install virt-customize sparrowdo git curl
|
||||
|
||||
# Verify libvirt is running
|
||||
sudo systemctl status libvirtd
|
||||
|
||||
# Verify SSH keys exist
|
||||
ls -la ~/.ssh/id_rsa ~/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 1: Set Variables
|
||||
|
||||
```bash
|
||||
# Set your configuration
|
||||
export BUILD_ID="manual-$(date +%s)"
|
||||
export IMAGES_DIR="/var/lib/libvirt/images"
|
||||
export WORK_DIR="/Users/ssimpson/Documents/git/resf-test-jenkins/manual-test-${BUILD_ID}"
|
||||
export SCRIPT_DIR="/Users/ssimpson/Documents/git/resf-test-jenkins"
|
||||
|
||||
# QCOW2 Image URL
|
||||
export QCOW2_URL="https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2"
|
||||
|
||||
# SSH Keys
|
||||
export SSH_PRIVATE_KEY="${HOME}/.ssh/id_rsa"
|
||||
export SSH_PUBLIC_KEY="${HOME}/.ssh/id_rsa.pub"
|
||||
|
||||
# Test Configuration - EDIT THESE
|
||||
export TEST_NAME="my-test"
|
||||
export TEST_REPO_URL="https://github.com/your-org/your-test-repo.git"
|
||||
export TEST_REPO_BRANCH="main"
|
||||
|
||||
# Image download behavior (set to "true" to force re-download)
|
||||
export REDOWNLOAD_IMAGE="false"
|
||||
|
||||
echo "Configuration set for build: ${BUILD_ID}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Initialize Environment
|
||||
|
||||
```bash
|
||||
# Create working directory
|
||||
mkdir -p "${WORK_DIR}"
|
||||
cd "${WORK_DIR}"
|
||||
|
||||
# Ensure images directory exists and is writable
|
||||
sudo mkdir -p "${IMAGES_DIR}"
|
||||
sudo chown ${USER}:${USER} "${IMAGES_DIR}"
|
||||
|
||||
# Verify scripts are executable
|
||||
chmod +x "${SCRIPT_DIR}/scripts"/*.sh
|
||||
|
||||
echo "Environment initialized"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Download Base QCOW2 Image
|
||||
|
||||
```bash
|
||||
# Extract filename from URL for caching
|
||||
export IMAGE_FILENAME=$(basename "${QCOW2_URL}")
|
||||
export CACHED_IMAGE="${IMAGES_DIR}/${IMAGE_FILENAME}"
|
||||
export BASE_IMAGE="${IMAGES_DIR}/base-${BUILD_ID}.qcow2"
|
||||
|
||||
echo "Image URL: ${QCOW2_URL}"
|
||||
echo "Cached image: ${CACHED_IMAGE}"
|
||||
echo "Build image: ${BASE_IMAGE}"
|
||||
|
||||
if [ "${REDOWNLOAD_IMAGE}" = "true" ]; then
|
||||
echo "REDOWNLOAD_IMAGE is enabled - forcing fresh download"
|
||||
curl -L --progress-bar -o "${BASE_IMAGE}" "${QCOW2_URL}"
|
||||
else
|
||||
if [ -f "${CACHED_IMAGE}" ]; then
|
||||
echo "Found cached image, creating copy for this build..."
|
||||
cp "${CACHED_IMAGE}" "${BASE_IMAGE}"
|
||||
else
|
||||
echo "No cached image found, downloading..."
|
||||
curl -L --progress-bar -o "${CACHED_IMAGE}" "${QCOW2_URL}"
|
||||
echo "Creating copy for this build..."
|
||||
cp "${CACHED_IMAGE}" "${BASE_IMAGE}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Verify the image
|
||||
echo "Verifying image..."
|
||||
qemu-img info "${BASE_IMAGE}" | head -10
|
||||
|
||||
echo "Base image ready: ${BASE_IMAGE}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4: Prepare Golden Image
|
||||
|
||||
```bash
|
||||
# Set golden image path
|
||||
export GOLDEN_IMAGE="${IMAGES_DIR}/golden-${BUILD_ID}.qcow2"
|
||||
|
||||
# Set prep script (you can customize this)
|
||||
export PREP_SCRIPT="${SCRIPT_DIR}/docs/default-prep.sh"
|
||||
|
||||
echo "Creating golden image..."
|
||||
echo "Base: ${BASE_IMAGE}"
|
||||
echo "Golden: ${GOLDEN_IMAGE}"
|
||||
echo "Prep Script: ${PREP_SCRIPT}"
|
||||
echo "SSH Key: ${SSH_PUBLIC_KEY}"
|
||||
|
||||
# Run the setup script
|
||||
"${SCRIPT_DIR}/scripts/setup_base.sh" \
|
||||
"${BASE_IMAGE}" \
|
||||
"${PREP_SCRIPT}" \
|
||||
"${GOLDEN_IMAGE}" \
|
||||
"${SSH_PUBLIC_KEY}"
|
||||
|
||||
echo "Golden image ready: ${GOLDEN_IMAGE}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 5: Provision Test VM
|
||||
|
||||
```bash
|
||||
# Set VM name
|
||||
export VM_NAME="${TEST_NAME}-${BUILD_ID}"
|
||||
|
||||
echo "Provisioning VM: ${VM_NAME}"
|
||||
|
||||
# Provision the VM and capture its IP
|
||||
export VM_IP=$("${SCRIPT_DIR}/scripts/provision_vm.sh" "${VM_NAME}" "${GOLDEN_IMAGE}" 60)
|
||||
|
||||
if [ "$VM_IP" = "ERROR" ] || [ -z "$VM_IP" ]; then
|
||||
echo "ERROR: Failed to provision VM"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "VM provisioned successfully"
|
||||
echo "VM Name: ${VM_NAME}"
|
||||
echo "IP Address: ${VM_IP}"
|
||||
|
||||
# Wait for SSH to be ready
|
||||
echo "Waiting for SSH to be ready..."
|
||||
for i in {1..30}; do
|
||||
if ssh -i "${SSH_PRIVATE_KEY}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
-o ConnectTimeout=5 \
|
||||
-o UserKnownHostsFile=/dev/null \
|
||||
root@${VM_IP} 'echo "SSH ready"' 2>/dev/null; then
|
||||
echo "SSH connection established!"
|
||||
break
|
||||
fi
|
||||
echo "Attempt $i/30..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# Test SSH connection manually
|
||||
echo "Testing SSH connection..."
|
||||
ssh -i "${SSH_PRIVATE_KEY}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
-o UserKnownHostsFile=/dev/null \
|
||||
root@${VM_IP} 'hostname && cat /etc/rocky-release'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 6: Clone Test Repository
|
||||
|
||||
```bash
|
||||
# Create test workspace
|
||||
export TEST_WORKSPACE="${WORK_DIR}/test-workspace"
|
||||
mkdir -p "${TEST_WORKSPACE}"
|
||||
cd "${TEST_WORKSPACE}"
|
||||
|
||||
echo "Cloning test repository..."
|
||||
echo "URL: ${TEST_REPO_URL}"
|
||||
echo "Branch: ${TEST_REPO_BRANCH}"
|
||||
|
||||
# Clone the repository
|
||||
git clone -b "${TEST_REPO_BRANCH}" "${TEST_REPO_URL}" test-repo
|
||||
|
||||
# Find the sparrowfile
|
||||
export SPARROWFILE=$(find test-repo -name sparrowfile -type f | head -1)
|
||||
|
||||
if [ -z "$SPARROWFILE" ]; then
|
||||
echo "ERROR: No sparrowfile found in repository"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Found sparrowfile: ${SPARROWFILE}"
|
||||
|
||||
# Show sparrowfile contents
|
||||
echo "Sparrowfile contents:"
|
||||
cat "${SPARROWFILE}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 7: Run Sparrowdo Test
|
||||
|
||||
```bash
|
||||
# Create logs directory
|
||||
mkdir -p "${TEST_WORKSPACE}/logs"
|
||||
|
||||
echo "=========================================="
|
||||
echo "Running Sparrowdo Test"
|
||||
echo "=========================================="
|
||||
echo "Target: root@${VM_IP}"
|
||||
echo "Sparrowfile: ${SPARROWFILE}"
|
||||
echo "Log: ${TEST_WORKSPACE}/logs/test.log"
|
||||
echo ""
|
||||
|
||||
# Run the test
|
||||
timeout 900 sparrowdo \
|
||||
--host="${VM_IP}" \
|
||||
--ssh_user=root \
|
||||
--ssh_private_key="${SSH_PRIVATE_KEY}" \
|
||||
--ssh_args="-o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null" \
|
||||
--no_sudo \
|
||||
--sparrowfile="${SPARROWFILE}" 2>&1 | tee "${TEST_WORKSPACE}/logs/test.log"
|
||||
|
||||
echo ""
|
||||
echo "Test completed!"
|
||||
echo "Logs saved to: ${TEST_WORKSPACE}/logs/test.log"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 8: Cleanup
|
||||
|
||||
```bash
|
||||
echo "=========================================="
|
||||
echo "Cleanup"
|
||||
echo "=========================================="
|
||||
|
||||
# Destroy the VM
|
||||
echo "Destroying VM: ${VM_NAME}"
|
||||
"${SCRIPT_DIR}/scripts/cleanup_vm.sh" "${VM_NAME}"
|
||||
|
||||
# Optional: Remove temporary images
|
||||
echo ""
|
||||
echo "Temporary images:"
|
||||
echo " Base: ${BASE_IMAGE}"
|
||||
echo " Golden: ${GOLDEN_IMAGE}"
|
||||
echo ""
|
||||
read -p "Remove temporary images? (y/n): " -n 1 -r
|
||||
echo
|
||||
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "Removing images..."
|
||||
sudo rm -f "${BASE_IMAGE}"
|
||||
sudo rm -f "${GOLDEN_IMAGE}"
|
||||
echo "Images removed"
|
||||
else
|
||||
echo "Images preserved at:"
|
||||
echo " ${BASE_IMAGE}"
|
||||
echo " ${GOLDEN_IMAGE}"
|
||||
fi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 9: View Results
|
||||
|
||||
```bash
|
||||
echo "=========================================="
|
||||
echo "Test Run Summary"
|
||||
echo "=========================================="
|
||||
echo "Build ID: ${BUILD_ID}"
|
||||
echo "Test Name: ${TEST_NAME}"
|
||||
echo "VM Name: ${VM_NAME}"
|
||||
echo "Working Directory: ${WORK_DIR}"
|
||||
echo "Test Logs: ${TEST_WORKSPACE}/logs/test.log"
|
||||
echo ""
|
||||
|
||||
# View the test log
|
||||
echo "=== Test Log ==="
|
||||
cat "${TEST_WORKSPACE}/logs/test.log"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting During Manual Run
|
||||
|
||||
### VM Won't Start
|
||||
```bash
|
||||
# Check libvirt status
|
||||
sudo systemctl status libvirtd
|
||||
|
||||
# List all VMs
|
||||
virsh list --all
|
||||
|
||||
# Check VM details
|
||||
virsh dominfo "${VM_NAME}"
|
||||
|
||||
# View VM console
|
||||
virsh console "${VM_NAME}" # Ctrl+] to exit
|
||||
```
|
||||
|
||||
### Can't Get IP Address
|
||||
```bash
|
||||
# Check network
|
||||
virsh net-list --all
|
||||
virsh net-info default
|
||||
|
||||
# View DHCP leases
|
||||
virsh net-dhcp-leases default
|
||||
|
||||
# Try alternative IP detection
|
||||
virsh domifaddr "${VM_NAME}" --source lease
|
||||
virsh domifaddr "${VM_NAME}" --source agent
|
||||
virsh domifaddr "${VM_NAME}" --source arp
|
||||
```
|
||||
|
||||
### SSH Connection Fails
|
||||
```bash
|
||||
# Test SSH manually with verbose output
|
||||
ssh -vvv -i "${SSH_PRIVATE_KEY}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
-o UserKnownHostsFile=/dev/null \
|
||||
root@${VM_IP}
|
||||
|
||||
# Check if SSH key was injected
|
||||
sudo virt-cat -a "${GOLDEN_IMAGE}" /root/.ssh/authorized_keys
|
||||
|
||||
# Check VM network from inside
|
||||
virsh console "${VM_NAME}"
|
||||
# Then inside VM:
|
||||
# ip addr
|
||||
# systemctl status sshd
|
||||
# firewall-cmd --list-all
|
||||
```
|
||||
|
||||
### Sparrowdo Fails
|
||||
```bash
|
||||
# Test connectivity first
|
||||
ssh -i "${SSH_PRIVATE_KEY}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
root@${VM_IP} 'which perl && perl --version'
|
||||
|
||||
# Run sparrowdo with debug output
|
||||
sparrowdo \
|
||||
--host="${VM_IP}" \
|
||||
--ssh_user=root \
|
||||
--ssh_private_key="${SSH_PRIVATE_KEY}" \
|
||||
--verbose \
|
||||
--sparrowfile="${SPARROWFILE}"
|
||||
```
|
||||
|
||||
### Clean Up Failed VMs
|
||||
```bash
|
||||
# List all VMs
|
||||
virsh list --all
|
||||
|
||||
# Force cleanup
|
||||
for vm in $(virsh list --all --name | grep "${BUILD_ID}"); do
|
||||
virsh destroy "$vm" 2>/dev/null || true
|
||||
virsh undefine "$vm" 2>/dev/null || true
|
||||
sudo rm -f "/var/lib/libvirt/images/${vm}.qcow2"
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Cleanup Script
|
||||
|
||||
If something goes wrong and you need to clean up everything:
|
||||
|
||||
```bash
|
||||
# Save this as cleanup-all.sh
|
||||
#!/bin/bash
|
||||
|
||||
BUILD_ID="${1:-manual}"
|
||||
|
||||
echo "Cleaning up build: ${BUILD_ID}"
|
||||
|
||||
# Stop and remove VMs
|
||||
for vm in $(virsh list --all --name | grep "${BUILD_ID}"); do
|
||||
echo "Removing VM: $vm"
|
||||
virsh destroy "$vm" 2>/dev/null || true
|
||||
virsh undefine "$vm" 2>/dev/null || true
|
||||
sudo rm -f "/var/lib/libvirt/images/${vm}.qcow2"
|
||||
done
|
||||
|
||||
# Remove images
|
||||
sudo rm -f "/var/lib/libvirt/images/base-${BUILD_ID}.qcow2"
|
||||
sudo rm -f "/var/lib/libvirt/images/golden-${BUILD_ID}.qcow2"
|
||||
|
||||
# Remove working directory
|
||||
rm -rf "/Users/ssimpson/Documents/git/resf-test-jenkins/manual-test-${BUILD_ID}"
|
||||
|
||||
echo "Cleanup complete"
|
||||
```
|
||||
|
||||
Then run:
|
||||
```bash
|
||||
chmod +x cleanup-all.sh
|
||||
./cleanup-all.sh manual-1234567890 # Use your BUILD_ID
|
||||
```
|
||||
@@ -1,361 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Manual Test Run - Simulates Jenkins Pipeline Steps
|
||||
# This script walks through all the steps Jenkins would take
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
GREEN='\033[0;32m'
|
||||
BLUE='\033[0;34m'
|
||||
YELLOW='\033[1;33m'
|
||||
RED='\033[0;31m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo_step() {
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
echo -e "${BLUE}$1${NC}"
|
||||
echo -e "${BLUE}========================================${NC}"
|
||||
}
|
||||
|
||||
echo_info() {
|
||||
echo -e "${GREEN}[INFO]${NC} $1"
|
||||
}
|
||||
|
||||
echo_warn() {
|
||||
echo -e "${YELLOW}[WARN]${NC} $1"
|
||||
}
|
||||
|
||||
echo_error() {
|
||||
echo -e "${RED}[ERROR]${NC} $1"
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# CONFIGURATION - Edit these variables
|
||||
# =============================================================================
|
||||
|
||||
# Build ID (simulating Jenkins BUILD_NUMBER)
|
||||
BUILD_ID="manual-$(date +%s)"
|
||||
|
||||
# QCOW2 Image URL
|
||||
QCOW2_URL="https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2"
|
||||
|
||||
# Image storage directory
|
||||
IMAGES_DIR="/var/lib/libvirt/images"
|
||||
|
||||
# Image download behavior
|
||||
# Set to "true" to force re-download even if cached image exists
|
||||
REDOWNLOAD_IMAGE="false"
|
||||
|
||||
# SSH Keys
|
||||
SSH_PRIVATE_KEY="${HOME}/.ssh/id_rsa"
|
||||
SSH_PUBLIC_KEY="${HOME}/.ssh/id_rsa.pub"
|
||||
|
||||
# Test repository configuration
|
||||
# Edit this to point to your actual test repository
|
||||
TEST_NAME="example-test"
|
||||
TEST_REPO_URL="https://github.com/your-org/your-test-repo.git"
|
||||
TEST_REPO_BRANCH="main"
|
||||
|
||||
# Working directory (this script's location)
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
|
||||
WORK_DIR="${SCRIPT_DIR}/manual-test-${BUILD_ID}"
|
||||
|
||||
# Golden image prep script location
|
||||
PREP_SCRIPT="${SCRIPT_DIR}/docs/default-prep.sh"
|
||||
|
||||
# =============================================================================
|
||||
# Step 1: Initialize
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 1: Initialize Environment"
|
||||
|
||||
echo_info "Build ID: ${BUILD_ID}"
|
||||
echo_info "Working Directory: ${WORK_DIR}"
|
||||
echo_info "Images Directory: ${IMAGES_DIR}"
|
||||
echo_info "Scripts Directory: ${SCRIPT_DIR}/scripts"
|
||||
|
||||
# Create working directory
|
||||
mkdir -p "${WORK_DIR}"
|
||||
cd "${WORK_DIR}"
|
||||
|
||||
# Ensure images directory exists
|
||||
sudo mkdir -p "${IMAGES_DIR}"
|
||||
sudo chown ${USER}:${USER} "${IMAGES_DIR}"
|
||||
|
||||
# Make scripts executable
|
||||
chmod +x "${SCRIPT_DIR}/scripts"/*.sh
|
||||
|
||||
echo_info "Initialization complete"
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 2..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 2: Download Base Image
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 2: Download Base QCOW2 Image"
|
||||
|
||||
# Extract filename from URL for caching
|
||||
IMAGE_FILENAME=$(basename "${QCOW2_URL}")
|
||||
CACHED_IMAGE="${IMAGES_DIR}/${IMAGE_FILENAME}"
|
||||
BASE_IMAGE="${IMAGES_DIR}/base-${BUILD_ID}.qcow2"
|
||||
|
||||
echo_info "Image URL: ${QCOW2_URL}"
|
||||
echo_info "Cached image path: ${CACHED_IMAGE}"
|
||||
echo_info "Build-specific image: ${BASE_IMAGE}"
|
||||
echo_info "Redownload setting: ${REDOWNLOAD_IMAGE}"
|
||||
echo ""
|
||||
|
||||
if [ "${REDOWNLOAD_IMAGE}" = "true" ]; then
|
||||
echo_warn "REDOWNLOAD_IMAGE is enabled - forcing fresh download"
|
||||
echo_info "Downloading from: ${QCOW2_URL}"
|
||||
curl -L --progress-bar -o "${BASE_IMAGE}" "${QCOW2_URL}"
|
||||
else
|
||||
if [ -f "${CACHED_IMAGE}" ]; then
|
||||
echo_info "Found cached image: ${CACHED_IMAGE}"
|
||||
echo_info "Creating copy for build ${BUILD_ID}..."
|
||||
cp "${CACHED_IMAGE}" "${BASE_IMAGE}"
|
||||
else
|
||||
echo_info "No cached image found at: ${CACHED_IMAGE}"
|
||||
echo_info "Downloading from: ${QCOW2_URL}"
|
||||
curl -L --progress-bar -o "${CACHED_IMAGE}" "${QCOW2_URL}"
|
||||
echo_info "Creating copy for build ${BUILD_ID}..."
|
||||
cp "${CACHED_IMAGE}" "${BASE_IMAGE}"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo_info "Verifying image..."
|
||||
qemu-img info "${BASE_IMAGE}" | head -10
|
||||
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 3..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 3: Prepare Golden Image
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 3: Prepare Golden Image"
|
||||
|
||||
GOLDEN_IMAGE="${IMAGES_DIR}/golden-${BUILD_ID}.qcow2"
|
||||
|
||||
echo_info "Golden image will be created at: ${GOLDEN_IMAGE}"
|
||||
echo_info "Using prep script: ${PREP_SCRIPT}"
|
||||
echo_info "Using SSH public key: ${SSH_PUBLIC_KEY}"
|
||||
|
||||
# Check if prep script exists
|
||||
if [ ! -f "${PREP_SCRIPT}" ]; then
|
||||
echo_error "Prep script not found: ${PREP_SCRIPT}"
|
||||
echo_info "You can create it or use the default one in docs/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if SSH keys exist
|
||||
if [ ! -f "${SSH_PUBLIC_KEY}" ]; then
|
||||
echo_error "SSH public key not found: ${SSH_PUBLIC_KEY}"
|
||||
echo_info "Generate one with: ssh-keygen -t rsa -b 4096 -f ${HOME}/.ssh/id_rsa"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo_info "Running setup_base.sh..."
|
||||
"${SCRIPT_DIR}/scripts/setup_base.sh" \
|
||||
"${BASE_IMAGE}" \
|
||||
"${PREP_SCRIPT}" \
|
||||
"${GOLDEN_IMAGE}" \
|
||||
"${SSH_PUBLIC_KEY}"
|
||||
|
||||
echo_info "Golden image ready: ${GOLDEN_IMAGE}"
|
||||
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 4 (Bootstrap)..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 4: Bootstrap Golden Image
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 4: Bootstrap Golden Image with Sparrowdo"
|
||||
|
||||
echo_info "Bootstrapping installs Sparrowdo and dependencies ONCE in the golden image"
|
||||
echo_info "This allows all test VMs to skip bootstrap and run tests immediately"
|
||||
echo ""
|
||||
|
||||
"${SCRIPT_DIR}/scripts/bootstrap_golden.sh" "${GOLDEN_IMAGE}" "${SSH_PRIVATE_KEY}"
|
||||
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 5 (Provision Test VM)..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 5: Provision Test VM
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 5: Provision Test VM"
|
||||
|
||||
VM_NAME="${TEST_NAME}-${BUILD_ID}"
|
||||
|
||||
echo_info "Creating VM: ${VM_NAME}"
|
||||
echo_info "Using golden image: ${GOLDEN_IMAGE}"
|
||||
|
||||
# Provision VM and capture output
|
||||
PROVISION_OUTPUT=$("${SCRIPT_DIR}/scripts/provision_vm.sh" "${VM_NAME}" "${GOLDEN_IMAGE}" 60)
|
||||
|
||||
# Extract just the IP address (last line of output)
|
||||
VM_IP=$(echo "$PROVISION_OUTPUT" | tail -1)
|
||||
|
||||
if [ "$VM_IP" = "ERROR" ] || [ -z "$VM_IP" ]; then
|
||||
echo_error "Failed to provision VM"
|
||||
echo_error "Provision output:"
|
||||
echo "$PROVISION_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo_info "VM provisioned successfully"
|
||||
echo_info "IP Address: ${VM_IP}"
|
||||
|
||||
# Wait for SSH to be ready (test with rocky user)
|
||||
echo_info "Waiting for SSH to be ready (testing with rocky user)..."
|
||||
for i in {1..30}; do
|
||||
if ssh -i "${SSH_PRIVATE_KEY}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
-o ConnectTimeout=5 \
|
||||
-o UserKnownHostsFile=/dev/null \
|
||||
rocky@${VM_IP} 'echo "SSH ready"' 2>/dev/null; then
|
||||
echo_info "SSH connection established for rocky user"
|
||||
break
|
||||
fi
|
||||
echo_info "Attempt $i/30..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 6 (Clone Test Repository)..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 6: Clone Test Repository
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 6: Clone Test Repository"
|
||||
|
||||
echo_info "Cloning: ${TEST_REPO_URL}"
|
||||
echo_info "Branch: ${TEST_REPO_BRANCH}"
|
||||
|
||||
# Create test workspace
|
||||
TEST_WORKSPACE="${WORK_DIR}/test-workspace"
|
||||
mkdir -p "${TEST_WORKSPACE}"
|
||||
cd "${TEST_WORKSPACE}"
|
||||
|
||||
# Clone the test repository
|
||||
if git clone -b "${TEST_REPO_BRANCH}" "${TEST_REPO_URL}" test-repo; then
|
||||
echo_info "Repository cloned successfully"
|
||||
else
|
||||
echo_error "Failed to clone repository"
|
||||
echo_warn "Cleanup will still run. Press Ctrl+C to abort or Enter to continue cleanup..."
|
||||
read
|
||||
"${SCRIPT_DIR}/scripts/cleanup_vm.sh" "${VM_NAME}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Find sparrowfile (look for main.raku or sparrowfile)
|
||||
SPARROWFILE=$(find test-repo -name main.raku -type f | head -1)
|
||||
if [ -z "$SPARROWFILE" ]; then
|
||||
SPARROWFILE=$(find test-repo -name sparrowfile -type f | head -1)
|
||||
fi
|
||||
|
||||
if [ -z "$SPARROWFILE" ]; then
|
||||
echo_error "No sparrowfile or main.raku found in repository"
|
||||
echo_warn "Cleanup will still run. Press Ctrl+C to abort or Enter to continue cleanup..."
|
||||
read
|
||||
"${SCRIPT_DIR}/scripts/cleanup_vm.sh" "${VM_NAME}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo_info "Found sparrowfile: ${SPARROWFILE}"
|
||||
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 7 (Run Test)..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 7: Run Sparrowdo Test
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 7: Run Sparrowdo Test"
|
||||
|
||||
# Create logs directory
|
||||
mkdir -p logs
|
||||
|
||||
echo_info "Running Sparrowdo test against VM..."
|
||||
echo_info "Target: rocky@${VM_IP}"
|
||||
echo_info "Sparrowfile: ${SPARROWFILE}"
|
||||
|
||||
# Run sparrowdo test
|
||||
if timeout 900 sparrowdo \
|
||||
--host="${VM_IP}" \
|
||||
--ssh_user=rocky \
|
||||
--ssh_private_key="${SSH_PRIVATE_KEY}" \
|
||||
--ssh_args="-o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null" \
|
||||
--no_sudo \
|
||||
--sparrowfile="${SPARROWFILE}" \
|
||||
--verbose \
|
||||
--color 2>&1 | tee logs/test.log; then
|
||||
echo_info "Test completed successfully!"
|
||||
else
|
||||
echo_error "Test failed or timed out"
|
||||
echo_info "Logs saved to: ${TEST_WORKSPACE}/logs/test.log"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo_info "Test logs location: ${TEST_WORKSPACE}/logs/test.log"
|
||||
|
||||
echo ""
|
||||
read -p "Press Enter to continue to Step 8 (Cleanup)..."
|
||||
|
||||
# =============================================================================
|
||||
# Step 8: Cleanup
|
||||
# =============================================================================
|
||||
|
||||
echo_step "Step 8: Cleanup"
|
||||
|
||||
echo_info "Cleaning up VM: ${VM_NAME}"
|
||||
"${SCRIPT_DIR}/scripts/cleanup_vm.sh" "${VM_NAME}"
|
||||
|
||||
echo ""
|
||||
echo_warn "Do you want to remove temporary images? (y/n)"
|
||||
echo_info "Base image: ${BASE_IMAGE}"
|
||||
echo_info "Golden image: ${GOLDEN_IMAGE}"
|
||||
read -p "Remove images? (y/n): " -n 1 -r
|
||||
echo
|
||||
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo_info "Removing temporary images..."
|
||||
sudo rm -f "${BASE_IMAGE}"
|
||||
sudo rm -f "${GOLDEN_IMAGE}"
|
||||
echo_info "Images removed"
|
||||
else
|
||||
echo_info "Images preserved for inspection"
|
||||
echo_info "Base image: ${BASE_IMAGE}"
|
||||
echo_info "Golden image: ${GOLDEN_IMAGE}"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# Summary
|
||||
# =============================================================================
|
||||
|
||||
echo ""
|
||||
echo_step "Summary"
|
||||
|
||||
echo_info "Build ID: ${BUILD_ID}"
|
||||
echo_info "Test Name: ${TEST_NAME}"
|
||||
echo_info "Working Directory: ${WORK_DIR}"
|
||||
echo_info "Test Logs: ${TEST_WORKSPACE}/logs/test.log"
|
||||
|
||||
if [ -f "${BASE_IMAGE}" ]; then
|
||||
echo_info "Base Image: ${BASE_IMAGE} (preserved)"
|
||||
fi
|
||||
|
||||
if [ -f "${GOLDEN_IMAGE}" ]; then
|
||||
echo_info "Golden Image: ${GOLDEN_IMAGE} (preserved)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo_info "Manual test run complete!"
|
||||
Reference in New Issue
Block a user