resf-testing-repo/docs/CHANGES.md

Recent Changes Summary

User Configuration Changes

Non-Root User (rocky)

• Changed from: root user for testing
• Changed to: rocky user (standard Rocky Linux user)
• Reason: Sparrowdo best practices require non-root user with sudo privileges

SSH Key Injection

• SSH keys are now injected for both root and rocky users
• The rocky user has NOPASSWD sudo access (required for Sparrowdo bootstrap)

Sparrowdo Bootstrap Step

• New requirement: Must run sparrowdo --bootstrap before running tests
• Bootstrap installs necessary Raku/Sparrowdo dependencies on the target VM
• Bootstrap runs with the rocky user

Test Execution Flow

Old Flow (Root User)

1. Provision VM
2. Wait for SSH (root user)
3. Run sparrowdo test (root user, --no_sudo)

New Flow (Rocky User with Bootstrap)

1. Provision VM
2. Wait for SSH (rocky user)
3. Run sparrowdo --bootstrap (rocky user)
4. Run sparrowdo test (rocky user, --no_sudo, --verbose, --color)

Sparrowfile Naming

• Tests can use either sparrowfile or main.raku
• The framework checks for main.raku first, then falls back to sparrowfile

Command Examples

Manual Bootstrap

sparrowdo \
  --host 192.168.124.54 \
  --ssh_user rocky \
  --ssh_private_key ~/.ssh/id_rsa \
  --bootstrap \
  --color

Manual Test Run

sparrowdo \
  --host 192.168.124.54 \
  --ssh_user rocky \
  --ssh_private_key ~/.ssh/id_rsa \
  --no_sudo \
  --sparrowfile test-repo/main.raku \
  --verbose \
  --color

Golden Image Changes

Users Created

1. rocky (primary test user)

   • Password: rockypass
   • Sudo: NOPASSWD:ALL
   • SSH key: Injected from ~/.ssh/id_rsa.pub

2. testuser (backward compatibility)

   • Password: testpass
   • Sudo: NOPASSWD:ALL
   • SSH key: Not injected

3. root

   • Password: rockytesting
   • SSH key: Injected from ~/.ssh/id_rsa.pub

Bug Fixes

1. virt-customize D-Bus Errors ✅

• Problem: Prep scripts used firewall-cmd, hostnamectl which don't work offline
• Solution: Removed D-Bus dependent commands from prep scripts
• Impact: Golden image creation now works reliably

2. VM Provisioning Script Failures ✅

• Problem: set -e caused script to exit on arithmetic operations
• Solution: Removed set -e, added explicit error handling
• Impact: VMs provision successfully and report proper errors

3. Virsh Connection Issues ✅

• Problem: Non-root users couldn't access libvirt without explicit URI
• Solution: Added -c qemu:///system to all virsh commands
• Impact: Scripts work for jenkins user and other non-root users

4. Image Caching ✅

• Problem: Framework re-downloaded QCOW2 images on every build
• Solution: Cache images by filename, copy for each build
• Impact: Massive time savings (2GB download → 2 second copy)
• Control: REDOWNLOAD_IMAGE parameter forces fresh download when needed

5. IP Address Capture in Manual Script ✅

• Problem: Script captured all output including progress messages
• Solution: Extract only the last line (the IP address)
• Impact: SSH connection works properly

File Changes

Modified Files

• Jenkinsfile - Added bootstrap step, rocky user, main.raku support
• scripts/setup_base.sh - Inject SSH keys for rocky user
• scripts/provision_vm.sh - Fixed set -e issue, improved error handling
• scripts/cleanup_vm.sh - Added explicit qemu:///system connection
• docs/default-prep.sh - Create rocky user, remove D-Bus commands
• docs/manual-test-run.sh - Add bootstrap step, use rocky user
• README.md - Updated prerequisites (guestfs-tools)

New Files

• docs/virt-customize-guide.md - Comprehensive guide on offline image customization
• docs/manual-steps.md - Step-by-step manual testing guide
• docs/CHANGES.md - This file

Testing Checklist

Before Running Tests

• SSH keys exist (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub)
• libvirt is running (sudo systemctl status libvirtd)
• Default network is active (virsh -c qemu:///system net-list)
• User has sudo access
• guestfs-tools installed (which virt-customize)
• Sparrowdo installed (which sparrowdo)

Golden Image Verification

# After creating golden image, verify users exist
sudo virt-customize -a /path/to/golden.qcow2 --run-command 'id rocky'
sudo virt-customize -a /path/to/golden.qcow2 --run-command 'cat /home/rocky/.ssh/authorized_keys'

Manual VM Test

# Provision a test VM
VM_IP=$(./scripts/provision_vm.sh test-vm-$(date +%s) /path/to/golden.qcow2 60)

# Test SSH as rocky user
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no rocky@$VM_IP

# Test sudo access
ssh -i ~/.ssh/id_rsa rocky@$VM_IP 'sudo whoami'

# Bootstrap Sparrowdo
sparrowdo --host $VM_IP --ssh_user rocky --bootstrap --color

# Clean up
./scripts/cleanup_vm.sh test-vm-XXXXX

Migration Notes

For Existing Tests

If you have existing Sparrowdo tests that assumed root user:

1. Update TEST_MATRIX in Jenkins to use rocky user
2. Ensure tests use --no_sudo flag
3. Add bootstrap step before test execution
4. Verify sudoers access if tests need elevated privileges

Jenkins Pipeline Changes

The Jenkinsfile automatically handles:

• Creating rocky user in golden image
• Injecting SSH keys for rocky user
• Running bootstrap before tests
• Using --no_sudo flag with rocky user

No manual intervention needed for Jenkins builds.

Performance Improvements

Image Caching

• First build: Downloads 2GB QCOW2 image (~5-10 minutes)
• Subsequent builds: Copies from cache (~2 seconds)
• Disk usage: One cached image + one per active build
• Cleanup: Cached images persist, build images auto-delete

VM Provisioning Speed

• Linked clones: New VM disk created in < 1 second
• Boot time: ~10-20 seconds to get IP address
• Total provision time: ~30 seconds from start to SSH ready

Troubleshooting

Bootstrap Fails

# Check if rocky user has sudo
ssh -i ~/.ssh/id_rsa rocky@VM_IP 'sudo whoami'

# Check if perl is installed
ssh -i ~/.ssh/id_rsa rocky@VM_IP 'which perl'

# Re-run bootstrap with verbose output
sparrowdo --host VM_IP --ssh_user rocky --bootstrap --color

Test Fails with Permission Denied

# Verify SSH key is injected
ssh -i ~/.ssh/id_rsa rocky@VM_IP 'cat ~/.ssh/authorized_keys'

# Check if using correct user
# Should be rocky@VM_IP not root@VM_IP

VM Won't Get IP

# Check network status
virsh -c qemu:///system net-list --all

# Start default network if needed
sudo virsh net-start default

# Check DHCP leases
virsh -c qemu:///system net-dhcp-leases default