mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-19 01:17:31 +00:00
fe65d81635
* feat(sync): add active-users command to monitor CLI Adds a new `active-users` command to the SuperSync monitor script that reports: - Total registered and verified user counts - Active users by time period (24h, 7d, 30d, 90d) based on device and sync activity - New registration counts - Recently active users table with device count and ops - Users who never synced Usage: npm run monitor -- active-users https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP * feat(sync): add engaged users metric to active-users report Shows users who were active on 3+ distinct days in the last 2 weeks with new sync operations, giving a measure of genuine recurring usage. https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP * docs(sync): add active-users to docker monitoring docs Add missing active-users command to docker-monitor.sh case statement, help text, and DOCKER-MONITORING.md guide. https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP * refactor(sync): improve active-users command from review feedback - Fix timezone-unsafe DATE(): use AT TIME ZONE 'UTC' with explicit double precision cast for consistent day boundaries - Replace correlated subquery with LEFT JOIN for ops_7d count - Show total active count when LIMIT truncates the table - Add --threshold and --limit CLI flags for flexibility - Combine device/ops metrics into single line (connected / syncing) - Add skipInQuick to run-all-monitoring.ts - Update docker-monitor.sh header, help text, and DOCKER-MONITORING.md with active-users command, flags, and performance notes - Clarify "never synced" output label https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP --------- Co-authored-by: Claude <noreply@anthropic.com>
298 lines
7.3 KiB
Markdown
298 lines
7.3 KiB
Markdown
# SuperSync Monitoring & Analysis Tools
|
|
|
|
Comprehensive suite of tools for monitoring and analyzing SuperSync server storage, operations, and user patterns.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Run all monitoring checks
|
|
npm run monitor:all
|
|
|
|
# Run quick health check (skip deep analysis)
|
|
npm run monitor:all:quick
|
|
|
|
# Save full report to file
|
|
npm run monitor:all:save
|
|
|
|
# Focus on specific user
|
|
npm run monitor:all -- --user 29
|
|
```
|
|
|
|
## Available Tools
|
|
|
|
### 1. Basic Monitoring (`monitor.ts`)
|
|
|
|
General server health and user storage tracking.
|
|
|
|
```bash
|
|
# System vitals (CPU, memory, disk, DB)
|
|
npm run monitor:dev -- stats
|
|
|
|
# Top 20 users by storage
|
|
npm run monitor:dev -- usage
|
|
|
|
# View usage history/trends
|
|
npm run monitor:dev -- usage-history --tail 20
|
|
|
|
# Active user counts and recent activity
|
|
npm run monitor:dev -- active-users
|
|
npm run monitor:dev -- active-users --threshold 5 --limit 50
|
|
|
|
# Recent operations analysis
|
|
npm run monitor:dev -- ops --tail 100
|
|
npm run monitor:dev -- ops --user 29
|
|
|
|
# View server logs
|
|
npm run monitor:dev -- logs --tail 200
|
|
npm run monitor:dev -- logs --search "error"
|
|
npm run monitor:dev -- logs --error
|
|
```
|
|
|
|
### 2. Storage Analysis (`analyze-storage.ts`)
|
|
|
|
Deep-dive analysis for investigating storage anomalies and patterns.
|
|
|
|
```bash
|
|
# Analyze operation size distribution
|
|
npm run analyze-storage -- operation-sizes
|
|
npm run analyze-storage -- operation-sizes --user 29
|
|
|
|
# Temporal patterns (bursts, daily/hourly trends)
|
|
npm run analyze-storage -- operation-timeline
|
|
npm run analyze-storage -- operation-timeline --user 29
|
|
|
|
# Breakdown by operation/entity types
|
|
npm run analyze-storage -- operation-types
|
|
npm run analyze-storage -- operation-types --user 29
|
|
|
|
# Find largest operations
|
|
npm run analyze-storage -- large-ops --limit 50
|
|
|
|
# Detect rapid-fire/sync loops (>5 ops/second by default)
|
|
npm run analyze-storage -- rapid-fire --threshold 10
|
|
|
|
# Analyze snapshot patterns
|
|
npm run analyze-storage -- snapshot-analysis
|
|
|
|
# Complete deep-dive for one user
|
|
npm run analyze-storage -- user-deep-dive --user 27
|
|
|
|
# Export operations to JSON for external analysis
|
|
npm run analyze-storage -- export-ops --user 29 --limit 1000
|
|
|
|
# Compare two users
|
|
npm run analyze-storage -- compare-users 27 29
|
|
```
|
|
|
|
### 3. Complete Monitoring Suite (`run-all-monitoring.ts`)
|
|
|
|
Runs all monitoring and analysis tools in sequence.
|
|
|
|
```bash
|
|
# Run everything
|
|
npm run monitor:all
|
|
|
|
# Quick mode (skip deep analysis)
|
|
npm run monitor:all:quick
|
|
|
|
# Save to timestamped file in monitoring-reports/
|
|
npm run monitor:all:save
|
|
|
|
# Focus on specific user
|
|
npm run monitor:all -- --user 29 --save
|
|
```
|
|
|
|
## Investigation Workflows
|
|
|
|
### Workflow 1: General Health Check
|
|
|
|
```bash
|
|
npm run monitor:all:quick
|
|
```
|
|
|
|
Review:
|
|
|
|
- System vitals
|
|
- Top users by storage
|
|
- Operation size distribution
|
|
- Large operations
|
|
- Rapid-fire detection
|
|
|
|
### Workflow 2: Investigate User with High Storage
|
|
|
|
User has unusually high storage (e.g., User #29 with 28k operations):
|
|
|
|
```bash
|
|
# Step 1: Get complete picture
|
|
npm run analyze-storage -- user-deep-dive --user 29
|
|
|
|
# Step 2: Check for rapid-fire patterns
|
|
npm run analyze-storage -- rapid-fire --threshold 3
|
|
|
|
# Step 3: Export for detailed analysis
|
|
npm run analyze-storage -- export-ops --user 29 --limit 5000
|
|
```
|
|
|
|
### Workflow 3: Investigate Large Operations
|
|
|
|
User has unusually large operations (e.g., User #27 with 54KB avg):
|
|
|
|
```bash
|
|
# Step 1: Find largest operations
|
|
npm run analyze-storage -- large-ops --limit 20
|
|
|
|
# Step 2: Analyze that user's patterns
|
|
npm run analyze-storage -- user-deep-dive --user 27
|
|
|
|
# Step 3: Compare with "normal" user
|
|
npm run analyze-storage -- compare-users 27 29
|
|
```
|
|
|
|
### Workflow 4: Investigate Sync Loops
|
|
|
|
Suspect a sync loop or rapid-fire operations:
|
|
|
|
```bash
|
|
# Step 1: Detect rapid-fire (lower threshold)
|
|
npm run analyze-storage -- rapid-fire --threshold 3
|
|
|
|
# Step 2: Timeline analysis for affected user
|
|
npm run analyze-storage -- operation-timeline --user 29
|
|
|
|
# Step 3: Check operation types
|
|
npm run analyze-storage -- operation-types --user 29
|
|
```
|
|
|
|
### Workflow 5: Monthly Report
|
|
|
|
Generate comprehensive monthly storage report:
|
|
|
|
```bash
|
|
# Generate and save full report
|
|
npm run monitor:all:save
|
|
|
|
# Review trends
|
|
npm run monitor:dev -- usage-history --tail 30
|
|
```
|
|
|
|
## Output Files
|
|
|
|
- **Usage History**: `logs/usage-history.jsonl` - Appended by `monitor.ts usage`
|
|
- **Analysis Exports**: `analysis-output/` - JSON exports from `export-ops`
|
|
- **Full Reports**: `monitoring-reports/` - Timestamped reports from `monitor:all --save`
|
|
|
|
## Common Patterns to Investigate
|
|
|
|
### High Operation Count (>10k ops)
|
|
|
|
Possible causes:
|
|
|
|
- Long-time user (check first_op timestamp)
|
|
- Sync loop (check rapid-fire detection)
|
|
- Small operations (check avg op size)
|
|
|
|
**Investigate**: `user-deep-dive`, `operation-timeline`, `rapid-fire`
|
|
|
|
### Large Average Operation Size (>10KB)
|
|
|
|
Possible causes:
|
|
|
|
- SYNC_IMPORT operations
|
|
- Large task attachments
|
|
- Bulk operations
|
|
|
|
**Investigate**: `large-ops`, `operation-types`, compare with normal users
|
|
|
|
### Many Operations per Second
|
|
|
|
Possible causes:
|
|
|
|
- Sync loop between devices
|
|
- Rapid user interaction
|
|
- Buggy client
|
|
|
|
**Investigate**: `rapid-fire`, `operation-timeline`, per-device breakdown in `user-deep-dive`
|
|
|
|
### Large Snapshots
|
|
|
|
Possible causes:
|
|
|
|
- High operation count triggering snapshot
|
|
- Large state size
|
|
|
|
**Investigate**: `snapshot-analysis`, correlation with op count
|
|
|
|
## Automation
|
|
|
|
You can set up cron jobs for regular monitoring:
|
|
|
|
```bash
|
|
# Daily health check at 2 AM
|
|
0 2 * * * cd /path/to/super-sync-server && npm run monitor:all:quick >> logs/daily-check.log 2>&1
|
|
|
|
# Weekly full report every Sunday at 3 AM
|
|
0 3 * * 0 cd /path/to/super-sync-server && npm run monitor:all:save
|
|
|
|
# Hourly rapid-fire detection
|
|
0 * * * * cd /path/to/super-sync-server && npm run analyze-storage -- rapid-fire >> logs/rapid-fire.log 2>&1
|
|
```
|
|
|
|
## Tips
|
|
|
|
1. **Start broad, then narrow**: Use `monitor:all:quick` first, then drill down with specific commands
|
|
2. **Always save significant findings**: Use `--save` or redirect output to files
|
|
3. **Compare users**: Use `compare-users` to understand what's "normal" vs anomalous
|
|
4. **Export for deep analysis**: Use `export-ops` to get raw data for custom analysis
|
|
5. **Watch trends**: Regular `usage-history` checks reveal growth patterns
|
|
|
|
## Troubleshooting
|
|
|
|
### "Database connection failed"
|
|
|
|
- Check DATABASE_URL in .env
|
|
- Ensure PostgreSQL is running
|
|
- Verify network access
|
|
|
|
### "Command not found: tsx"
|
|
|
|
- Install tsx globally: `npm install -g tsx`
|
|
- Or use npx: `npx tsx scripts/analyze-storage.ts ...`
|
|
|
|
### "Out of memory"
|
|
|
|
- Reduce `--limit` values
|
|
- Run in quick mode
|
|
- Increase Node.js heap: `NODE_OPTIONS=--max-old-space-size=4096 npm run ...`
|
|
|
|
### "Query timeout"
|
|
|
|
- Database might be under load
|
|
- Reduce time ranges
|
|
- Add indexes if needed
|
|
|
|
## Development
|
|
|
|
To add new analysis commands:
|
|
|
|
1. Add function to `scripts/analyze-storage.ts`
|
|
2. Add case to `main()` switch
|
|
3. Update `getMonitoringCommands()` in `run-all-monitoring.ts` if it should run in full suite
|
|
4. Document here
|
|
|
|
## Performance Notes
|
|
|
|
- **Quick mode**: ~10-30 seconds
|
|
- **Full suite**: ~1-3 minutes (depends on data size)
|
|
- **user-deep-dive**: ~5-15 seconds per user
|
|
- **export-ops**: ~1-5 seconds per 1000 operations
|
|
|
|
## Security Notes
|
|
|
|
- Exports contain full operation payloads - handle securely
|
|
- User emails are included in outputs - be mindful of privacy
|
|
- Encrypted payloads show as encrypted in analysis
|
|
- Clean up old reports periodically
|
|
|
|
---
|
|
|
|
**Questions or issues?** File an issue or check the main SuperSync documentation.
|