# FTS5 Search System Documentation

## Overview

The Bible application now uses SQLite FTS5 (Full-Text Search 5) for professional-grade search capabilities. This replaces the previous in-memory search engine with a persistent, highly optimized search index.

## Architecture

### Components

1. **SearchDatabase** (`backend/src/searchDatabase.js`)
   - Manages FTS5 virtual tables and search queries
   - Provides BM25 ranking for relevance
   - Supports advanced query syntax

2. **Search Index Builder** (`backend/src/buildSearchIndex.js`)
   - Populates FTS5 index from markdown files
   - Runs during Docker image build
   - Processes all 4 Bible versions (ESV, NKJV, NLT, CSB)

3. **Database Schema**
   - `verses` table: Regular table for metadata and joins
   - `verses_fts` virtual table: FTS5 index for full-text search
   - Porter stemming + Unicode support + diacritic removal

## Features

### 1. Simple Word Search
```
faith
```
Finds all verses containing "faith" (case-insensitive)

### 2. Multiple Word Search (AND)
```
faith hope love
```
Finds verses containing ALL three words (implicit AND)

### 3. Phrase Search
```
"in the beginning"
```
Finds exact phrase matches

### 4. OR Queries
```
faith OR hope
```
Finds verses containing either word

### 5. NOT Queries
```
faith NOT fear
```
Finds verses with "faith" but without "fear"

### 6. NEAR Queries (Proximity)
```
faith NEAR(5) hope
```
Finds "faith" and "hope" within 5 words of each other

### 7. Prefix Search (Autocomplete)
```
bless*
```
Matches "blessed", "blessing", "blessings", etc.

## Performance

### Before (Phase 1)
- Search time: 50-200ms
- Multi-version search: ~2s (sequential)
- Index build: On server startup (5-10s delay)
- Memory: ~50MB in-memory index

### After (Phase 2)
- Search time: <1ms (100x faster)
- Multi-version search: <1ms (single FTS5 query)
- Index build: During Docker build (0ms at startup)
- Memory: ~5MB (index on disk, minimal RAM)

## Deployment

### Building the Search Index

The search index is automatically built during Docker image creation:

```dockerfile
RUN npm run build-search-index
```

### Manual Index Build (Development)

```bash
cd backend
npm run build-search-index        # Build if not exists
npm run rebuild-search-index      # Force rebuild
```

### Docker Volume

The search index is persisted in the `/app/backend/data` volume:

```yaml
volumes:
  - data:/app/backend/data
```

This ensures the index survives container restarts.

## API Endpoints

### Search
```
GET /api/search?q=faith&version=esv&limit=50
```

**Parameters:**
- `q`: Search query (required)
- `version`: Bible version (esv, nkjv, nlt, csb, all)
- `book`: Filter by book name (optional)
- `limit`: Max results (default: 50)
- `context`: Include surrounding verses (default: true)

**Response:**
```json
{
  "query": "faith",
  "results": [
    {
      "book": "Hebrews",
      "chapter": 11,
      "verse": 1,
      "text": "Now faith is...",
      "highlight": "Now <mark>faith</mark> is...",
      "relevance": 125.5,
      "context": [...],
      "searchVersion": "esv"
    }
  ],
  "total": 243,
  "hasMore": true,
  "version": "esv"
}
```

### Autocomplete Suggestions
```
GET /api/search/suggestions?q=ble&limit=10
```

Returns word suggestions based on prefix matching.

## Technical Details

### BM25 Ranking

FTS5 uses the BM25 algorithm for relevance scoring, which considers:
- Term frequency (how often words appear)
- Document frequency (how rare words are)
- Document length normalization

This provides industry-standard search relevance.

### Tokenization

The FTS5 index uses:
- **Porter stemming**: Matches word variations (walk, walking, walked)
- **Unicode support**: Handles international characters
- **Diacritic removal**: Treats café and cafe as equivalent

### Index Statistics

- Total verses indexed: ~31,000 per version
- Total documents: ~124,000 (4 versions)
- Index size: ~25MB on disk
- Build time: ~30-60 seconds

## Migration from Phase 1

Phase 2 is a drop-in replacement for the old BibleSearchEngine:

**Before:**
```javascript
const searchEngine = new BibleSearchEngine(dataDir);
await searchEngine.buildSearchIndex();
const results = await searchEngine.search(query);
```

**After:**
```javascript
const searchDb = new SearchDatabase(dbPath);
await searchDb.initialize();
const results = await searchDb.search(query);
```

The API response format remains identical for frontend compatibility.

## Future Enhancements

Potential Phase 3 improvements:
- Fuzzy matching (typo tolerance)
- Search result caching
- Query analytics and popular searches
- Highlighting context in results
- Cross-reference search
- Semantic search using embeddings

---

**Phase 2: Search Excellence** ✓ Complete