# SuperSync Monitoring & Analysis Tools

Comprehensive suite of tools for monitoring and analyzing SuperSync server storage, operations, and user patterns.

## Quick Start

```bash
# Run all monitoring checks
npm run monitor:all

# Run quick health check (skip deep analysis)
npm run monitor:all:quick

# Save full report to file
npm run monitor:all:save

# Focus on specific user
npm run monitor:all -- --user 29
```

## Available Tools

### 1. Basic Monitoring (`monitor.ts`)

General server health and user storage tracking.

```bash
# System vitals (CPU, memory, disk, DB)
npm run monitor:dev -- stats

# Top 20 users by storage
npm run monitor:dev -- usage

# View usage history/trends
npm run monitor:dev -- usage-history --tail 20

# Active user counts and recent activity
npm run monitor:dev -- active-users
npm run monitor:dev -- active-users --threshold 5 --limit 50

# Recent operations analysis
npm run monitor:dev -- ops --tail 100
npm run monitor:dev -- ops --user 29

# View server logs
npm run monitor:dev -- logs --tail 200
npm run monitor:dev -- logs --search "error"
npm run monitor:dev -- logs --error
```

### 2. Storage Analysis (`analyze-storage.ts`)

Deep-dive analysis for investigating storage anomalies and patterns.

```bash
# Analyze operation size distribution
npm run analyze-storage -- operation-sizes
npm run analyze-storage -- operation-sizes --user 29

# Temporal patterns (bursts, daily/hourly trends)
npm run analyze-storage -- operation-timeline
npm run analyze-storage -- operation-timeline --user 29

# Breakdown by operation/entity types
npm run analyze-storage -- operation-types
npm run analyze-storage -- operation-types --user 29

# Find largest operations
npm run analyze-storage -- large-ops --limit 50

# Detect rapid-fire/sync loops (>5 ops/second by default)
npm run analyze-storage -- rapid-fire --threshold 10

# Analyze snapshot patterns
npm run analyze-storage -- snapshot-analysis

# Complete deep-dive for one user
npm run analyze-storage -- user-deep-dive --user 27

# Export operations to JSON for external analysis
npm run analyze-storage -- export-ops --user 29 --limit 1000

# Compare two users
npm run analyze-storage -- compare-users 27 29
```

### 3. Complete Monitoring Suite (`run-all-monitoring.ts`)

Runs all monitoring and analysis tools in sequence.

```bash
# Run everything
npm run monitor:all

# Quick mode (skip deep analysis)
npm run monitor:all:quick

# Save to timestamped file in monitoring-reports/
npm run monitor:all:save

# Focus on specific user
npm run monitor:all -- --user 29 --save
```

## Investigation Workflows

### Workflow 1: General Health Check

```bash
npm run monitor:all:quick
```

Review:

- System vitals
- Top users by storage
- Operation size distribution
- Large operations
- Rapid-fire detection

### Workflow 2: Investigate User with High Storage

User has unusually high storage (e.g., User #29 with 28k operations):

```bash
# Step 1: Get complete picture
npm run analyze-storage -- user-deep-dive --user 29

# Step 2: Check for rapid-fire patterns
npm run analyze-storage -- rapid-fire --threshold 3

# Step 3: Export for detailed analysis
npm run analyze-storage -- export-ops --user 29 --limit 5000
```

### Workflow 3: Investigate Large Operations

User has unusually large operations (e.g., User #27 with 54KB avg):

```bash
# Step 1: Find largest operations
npm run analyze-storage -- large-ops --limit 20

# Step 2: Analyze that user's patterns
npm run analyze-storage -- user-deep-dive --user 27

# Step 3: Compare with "normal" user
npm run analyze-storage -- compare-users 27 29
```

### Workflow 4: Investigate Sync Loops

Suspect a sync loop or rapid-fire operations:

```bash
# Step 1: Detect rapid-fire (lower threshold)
npm run analyze-storage -- rapid-fire --threshold 3

# Step 2: Timeline analysis for affected user
npm run analyze-storage -- operation-timeline --user 29

# Step 3: Check operation types
npm run analyze-storage -- operation-types --user 29
```

### Workflow 5: Monthly Report

Generate comprehensive monthly storage report:

```bash
# Generate and save full report
npm run monitor:all:save

# Review trends
npm run monitor:dev -- usage-history --tail 30
```

## Output Files

- **Usage History**: `logs/usage-history.jsonl` - Appended by `monitor.ts usage`
- **Analysis Exports**: `analysis-output/` - JSON exports from `export-ops`
- **Full Reports**: `monitoring-reports/` - Timestamped reports from `monitor:all --save`

## Common Patterns to Investigate

### High Operation Count (>10k ops)

Possible causes:

- Long-time user (check first_op timestamp)
- Sync loop (check rapid-fire detection)
- Small operations (check avg op size)

**Investigate**: `user-deep-dive`, `operation-timeline`, `rapid-fire`

### Large Average Operation Size (>10KB)

Possible causes:

- SYNC_IMPORT operations
- Large task attachments
- Bulk operations

**Investigate**: `large-ops`, `operation-types`, compare with normal users

### Many Operations per Second

Possible causes:

- Sync loop between devices
- Rapid user interaction
- Buggy client

**Investigate**: `rapid-fire`, `operation-timeline`, per-device breakdown in `user-deep-dive`

### Large Snapshots

Possible causes:

- High operation count triggering snapshot
- Large state size

**Investigate**: `snapshot-analysis`, correlation with op count

## Automation

You can set up cron jobs for regular monitoring:

```bash
# Daily health check at 2 AM
0 2 * * * cd /path/to/super-sync-server && npm run monitor:all:quick >> logs/daily-check.log 2>&1

# Weekly full report every Sunday at 3 AM
0 3 * * 0 cd /path/to/super-sync-server && npm run monitor:all:save

# Hourly rapid-fire detection
0 * * * * cd /path/to/super-sync-server && npm run analyze-storage -- rapid-fire >> logs/rapid-fire.log 2>&1
```

## Tips

1. **Start broad, then narrow**: Use `monitor:all:quick` first, then drill down with specific commands
2. **Always save significant findings**: Use `--save` or redirect output to files
3. **Compare users**: Use `compare-users` to understand what's "normal" vs anomalous
4. **Export for deep analysis**: Use `export-ops` to get raw data for custom analysis
5. **Watch trends**: Regular `usage-history` checks reveal growth patterns

## Troubleshooting

### "Database connection failed"

- Check DATABASE_URL in .env
- Ensure PostgreSQL is running
- Verify network access

### "Command not found: tsx"

- Install tsx globally: `npm install -g tsx`
- Or use npx: `npx tsx scripts/analyze-storage.ts ...`

### "Out of memory"

- Reduce `--limit` values
- Run in quick mode
- Increase Node.js heap: `NODE_OPTIONS=--max-old-space-size=4096 npm run ...`

### "Query timeout"

- Database might be under load
- Reduce time ranges
- Add indexes if needed

## Development

To add new analysis commands:

1. Add function to `scripts/analyze-storage.ts`
2. Add case to `main()` switch
3. Update `getMonitoringCommands()` in `run-all-monitoring.ts` if it should run in full suite
4. Document here

## Performance Notes

- **Quick mode**: ~10-30 seconds
- **Full suite**: ~1-3 minutes (depends on data size)
- **user-deep-dive**: ~5-15 seconds per user
- **export-ops**: ~1-5 seconds per 1000 operations

## Security Notes

- Exports contain full operation payloads - handle securely
- User emails are included in outputs - be mindful of privacy
- Encrypted payloads show as encrypted in analysis
- Clean up old reports periodically

---

**Questions or issues?** File an issue or check the main SuperSync documentation.
