super-productivity/packages/super-sync-server/scripts/MONITORING-README.md

SuperSync Monitoring & Analysis Tools

Comprehensive suite of tools for monitoring and analyzing SuperSync server storage, operations, and user patterns.

Quick Start

# Run all monitoring checks
npm run monitor:all

# Run quick health check (skip deep analysis)
npm run monitor:all:quick

# Save full report to file
npm run monitor:all:save

# Focus on specific user
npm run monitor:all -- --user 29

Available Tools

1. Basic Monitoring (monitor.ts)

General server health and user storage tracking.

# System vitals (CPU, memory, disk, DB)
npm run monitor:dev -- stats

# Top 20 users by storage
npm run monitor:dev -- usage

# View usage history/trends
npm run monitor:dev -- usage-history --tail 20

# Active user counts and recent activity
npm run monitor:dev -- active-users
npm run monitor:dev -- active-users --threshold 5 --limit 50

# Recent operations analysis
npm run monitor:dev -- ops --tail 100
npm run monitor:dev -- ops --user 29

# View server logs
npm run monitor:dev -- logs --tail 200
npm run monitor:dev -- logs --search "error"
npm run monitor:dev -- logs --error

2. Storage Analysis (analyze-storage.ts)

Deep-dive analysis for investigating storage anomalies and patterns.

# Analyze operation size distribution
npm run analyze-storage -- operation-sizes
npm run analyze-storage -- operation-sizes --user 29

# Temporal patterns (bursts, daily/hourly trends)
npm run analyze-storage -- operation-timeline
npm run analyze-storage -- operation-timeline --user 29

# Breakdown by operation/entity types
npm run analyze-storage -- operation-types
npm run analyze-storage -- operation-types --user 29

# Find largest operations
npm run analyze-storage -- large-ops --limit 50

# Detect rapid-fire/sync loops (>5 ops/second by default)
npm run analyze-storage -- rapid-fire --threshold 10

# Analyze snapshot patterns
npm run analyze-storage -- snapshot-analysis

# Complete deep-dive for one user
npm run analyze-storage -- user-deep-dive --user 27

# Export operations to JSON for external analysis
npm run analyze-storage -- export-ops --user 29 --limit 1000

# Compare two users
npm run analyze-storage -- compare-users 27 29

3. Complete Monitoring Suite (run-all-monitoring.ts)

Runs all monitoring and analysis tools in sequence.

# Run everything
npm run monitor:all

# Quick mode (skip deep analysis)
npm run monitor:all:quick

# Save to timestamped file in monitoring-reports/
npm run monitor:all:save

# Focus on specific user
npm run monitor:all -- --user 29 --save

Investigation Workflows

Workflow 1: General Health Check

npm run monitor:all:quick

Review:

• System vitals
• Top users by storage
• Operation size distribution
• Large operations
• Rapid-fire detection

Workflow 2: Investigate User with High Storage

User has unusually high storage (e.g., User #29 with 28k operations):

# Step 1: Get complete picture
npm run analyze-storage -- user-deep-dive --user 29

# Step 2: Check for rapid-fire patterns
npm run analyze-storage -- rapid-fire --threshold 3

# Step 3: Export for detailed analysis
npm run analyze-storage -- export-ops --user 29 --limit 5000

Workflow 3: Investigate Large Operations

User has unusually large operations (e.g., User #27 with 54KB avg):

# Step 1: Find largest operations
npm run analyze-storage -- large-ops --limit 20

# Step 2: Analyze that user's patterns
npm run analyze-storage -- user-deep-dive --user 27

# Step 3: Compare with "normal" user
npm run analyze-storage -- compare-users 27 29

Workflow 4: Investigate Sync Loops

Suspect a sync loop or rapid-fire operations:

# Step 1: Detect rapid-fire (lower threshold)
npm run analyze-storage -- rapid-fire --threshold 3

# Step 2: Timeline analysis for affected user
npm run analyze-storage -- operation-timeline --user 29

# Step 3: Check operation types
npm run analyze-storage -- operation-types --user 29

Workflow 5: Monthly Report

Generate comprehensive monthly storage report:

# Generate and save full report
npm run monitor:all:save

# Review trends
npm run monitor:dev -- usage-history --tail 30

Output Files

• Usage History: logs/usage-history.jsonl - Appended by monitor.ts usage
• Analysis Exports: analysis-output/ - JSON exports from export-ops
• Full Reports: monitoring-reports/ - Timestamped reports from monitor:all --save

Common Patterns to Investigate

High Operation Count (>10k ops)

Possible causes:

• Long-time user (check first_op timestamp)
• Sync loop (check rapid-fire detection)
• Small operations (check avg op size)

Investigate: user-deep-dive, operation-timeline, rapid-fire

Large Average Operation Size (>10KB)

Possible causes:

• SYNC_IMPORT operations
• Large task attachments
• Bulk operations

Investigate: large-ops, operation-types, compare with normal users

Many Operations per Second

Possible causes:

• Sync loop between devices
• Rapid user interaction
• Buggy client

Investigate: rapid-fire, operation-timeline, per-device breakdown in user-deep-dive

Large Snapshots

Possible causes:

• High operation count triggering snapshot
• Large state size

Investigate: snapshot-analysis, correlation with op count

Automation

You can set up cron jobs for regular monitoring:

# Daily health check at 2 AM
0 2 * * * cd /path/to/super-sync-server && npm run monitor:all:quick >> logs/daily-check.log 2>&1

# Weekly full report every Sunday at 3 AM
0 3 * * 0 cd /path/to/super-sync-server && npm run monitor:all:save

# Hourly rapid-fire detection
0 * * * * cd /path/to/super-sync-server && npm run analyze-storage -- rapid-fire >> logs/rapid-fire.log 2>&1

Tips

1. Start broad, then narrow: Use monitor:all:quick first, then drill down with specific commands
2. Always save significant findings: Use --save or redirect output to files
3. Compare users: Use compare-users to understand what's "normal" vs anomalous
4. Export for deep analysis: Use export-ops to get raw data for custom analysis
5. Watch trends: Regular usage-history checks reveal growth patterns

Troubleshooting

"Database connection failed"

• Check DATABASE_URL in .env
• Ensure PostgreSQL is running
• Verify network access

"Command not found: tsx"

• Install tsx globally: npm install -g tsx
• Or use npx: npx tsx scripts/analyze-storage.ts ...

"Out of memory"

• Reduce --limit values
• Run in quick mode
• Increase Node.js heap: NODE_OPTIONS=--max-old-space-size=4096 npm run ...

"Query timeout"

• Database might be under load
• Reduce time ranges
• Add indexes if needed

Development

To add new analysis commands:

1. Add function to scripts/analyze-storage.ts
2. Add case to main() switch
3. Update getMonitoringCommands() in run-all-monitoring.ts if it should run in full suite
4. Document here

Performance Notes

• Quick mode: ~10-30 seconds
• Full suite: ~1-3 minutes (depends on data size)
• user-deep-dive: ~5-15 seconds per user
• export-ops: ~1-5 seconds per 1000 operations

Security Notes

• Exports contain full operation payloads - handle securely
• User emails are included in outputs - be mindful of privacy
• Encrypted payloads show as encrypted in analysis
• Clean up old reports periodically

---

Questions or issues? File an issue or check the main SuperSync documentation.