# Recent Changes Summary

## User Configuration Changes

### Non-Root User (rocky)
- **Changed from**: `root` user for testing
- **Changed to**: `rocky` user (standard Rocky Linux user)
- **Reason**: Sparrowdo best practices require non-root user with sudo privileges

### SSH Key Injection
- SSH keys are now injected for both `root` and `rocky` users
- The `rocky` user has NOPASSWD sudo access (required for Sparrowdo bootstrap)

### Sparrowdo Bootstrap Step
- **New requirement**: Must run `sparrowdo --bootstrap` before running tests
- Bootstrap installs necessary Raku/Sparrowdo dependencies on the target VM
- Bootstrap runs with the `rocky` user

## Test Execution Flow

### Old Flow (Root User)
```bash
1. Provision VM
2. Wait for SSH (root user)
3. Run sparrowdo test (root user, --no_sudo)
```

### New Flow (Rocky User with Bootstrap)
```bash
1. Provision VM
2. Wait for SSH (rocky user)
3. Run sparrowdo --bootstrap (rocky user)
4. Run sparrowdo test (rocky user, --no_sudo, --verbose, --color)
```

## Sparrowfile Naming
- Tests can use either `sparrowfile` or `main.raku`
- The framework checks for `main.raku` first, then falls back to `sparrowfile`

## Command Examples

### Manual Bootstrap
```bash
sparrowdo \
  --host 192.168.124.54 \
  --ssh_user rocky \
  --ssh_private_key ~/.ssh/id_rsa \
  --bootstrap \
  --color
```

### Manual Test Run
```bash
sparrowdo \
  --host 192.168.124.54 \
  --ssh_user rocky \
  --ssh_private_key ~/.ssh/id_rsa \
  --no_sudo \
  --sparrowfile test-repo/main.raku \
  --verbose \
  --color
```

## Golden Image Changes

### Users Created
1. **rocky** (primary test user)
   - Password: `rockypass`
   - Sudo: NOPASSWD:ALL
   - SSH key: Injected from `~/.ssh/id_rsa.pub`

2. **testuser** (backward compatibility)
   - Password: `testpass`
   - Sudo: NOPASSWD:ALL
   - SSH key: Not injected

3. **root**
   - Password: `rockytesting`
   - SSH key: Injected from `~/.ssh/id_rsa.pub`

## Bug Fixes

### 1. virt-customize D-Bus Errors ✅
- **Problem**: Prep scripts used `firewall-cmd`, `hostnamectl` which don't work offline
- **Solution**: Removed D-Bus dependent commands from prep scripts
- **Impact**: Golden image creation now works reliably

### 2. VM Provisioning Script Failures ✅
- **Problem**: `set -e` caused script to exit on arithmetic operations
- **Solution**: Removed `set -e`, added explicit error handling
- **Impact**: VMs provision successfully and report proper errors

### 3. Virsh Connection Issues ✅
- **Problem**: Non-root users couldn't access libvirt without explicit URI
- **Solution**: Added `-c qemu:///system` to all virsh commands
- **Impact**: Scripts work for jenkins user and other non-root users

### 4. Image Caching ✅
- **Problem**: Framework re-downloaded QCOW2 images on every build
- **Solution**: Cache images by filename, copy for each build
- **Impact**: Massive time savings (2GB download → 2 second copy)
- **Control**: `REDOWNLOAD_IMAGE` parameter forces fresh download when needed

### 5. IP Address Capture in Manual Script ✅
- **Problem**: Script captured all output including progress messages
- **Solution**: Extract only the last line (the IP address)
- **Impact**: SSH connection works properly

## File Changes

### Modified Files
- `Jenkinsfile` - Added bootstrap step, rocky user, main.raku support
- `scripts/setup_base.sh` - Inject SSH keys for rocky user
- `scripts/provision_vm.sh` - Fixed set -e issue, improved error handling
- `scripts/cleanup_vm.sh` - Added explicit qemu:///system connection
- `docs/default-prep.sh` - Create rocky user, remove D-Bus commands
- `docs/manual-test-run.sh` - Add bootstrap step, use rocky user
- `README.md` - Updated prerequisites (guestfs-tools)

### New Files
- `docs/virt-customize-guide.md` - Comprehensive guide on offline image customization
- `docs/manual-steps.md` - Step-by-step manual testing guide
- `docs/CHANGES.md` - This file

## Testing Checklist

### Before Running Tests
- [ ] SSH keys exist (`~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`)
- [ ] libvirt is running (`sudo systemctl status libvirtd`)
- [ ] Default network is active (`virsh -c qemu:///system net-list`)
- [ ] User has sudo access
- [ ] guestfs-tools installed (`which virt-customize`)
- [ ] Sparrowdo installed (`which sparrowdo`)

### Golden Image Verification
```bash
# After creating golden image, verify users exist
sudo virt-customize -a /path/to/golden.qcow2 --run-command 'id rocky'
sudo virt-customize -a /path/to/golden.qcow2 --run-command 'cat /home/rocky/.ssh/authorized_keys'
```

### Manual VM Test
```bash
# Provision a test VM
VM_IP=$(./scripts/provision_vm.sh test-vm-$(date +%s) /path/to/golden.qcow2 60)

# Test SSH as rocky user
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no rocky@$VM_IP

# Test sudo access
ssh -i ~/.ssh/id_rsa rocky@$VM_IP 'sudo whoami'

# Bootstrap Sparrowdo
sparrowdo --host $VM_IP --ssh_user rocky --bootstrap --color

# Clean up
./scripts/cleanup_vm.sh test-vm-XXXXX
```

## Migration Notes

### For Existing Tests
If you have existing Sparrowdo tests that assumed root user:

1. **Update TEST_MATRIX** in Jenkins to use rocky user
2. **Ensure tests use --no_sudo** flag
3. **Add bootstrap step** before test execution
4. **Verify sudoers access** if tests need elevated privileges

### Jenkins Pipeline Changes
The Jenkinsfile automatically handles:
- Creating rocky user in golden image
- Injecting SSH keys for rocky user
- Running bootstrap before tests
- Using `--no_sudo` flag with rocky user

No manual intervention needed for Jenkins builds.

## Performance Improvements

### Image Caching
- **First build**: Downloads 2GB QCOW2 image (~5-10 minutes)
- **Subsequent builds**: Copies from cache (~2 seconds)
- **Disk usage**: One cached image + one per active build
- **Cleanup**: Cached images persist, build images auto-delete

### VM Provisioning Speed
- **Linked clones**: New VM disk created in < 1 second
- **Boot time**: ~10-20 seconds to get IP address
- **Total provision time**: ~30 seconds from start to SSH ready

## Troubleshooting

### Bootstrap Fails
```bash
# Check if rocky user has sudo
ssh -i ~/.ssh/id_rsa rocky@VM_IP 'sudo whoami'

# Check if perl is installed
ssh -i ~/.ssh/id_rsa rocky@VM_IP 'which perl'

# Re-run bootstrap with verbose output
sparrowdo --host VM_IP --ssh_user rocky --bootstrap --color
```

### Test Fails with Permission Denied
```bash
# Verify SSH key is injected
ssh -i ~/.ssh/id_rsa rocky@VM_IP 'cat ~/.ssh/authorized_keys'

# Check if using correct user
# Should be rocky@VM_IP not root@VM_IP
```

### VM Won't Get IP
```bash
# Check network status
virsh -c qemu:///system net-list --all

# Start default network if needed
sudo virsh net-start default

# Check DHCP leases
virsh -c qemu:///system net-dhcp-leases default
```