chore: remove webhook system

2025-07-29 15:52:03 +03:30
parent 3bf158c433
commit 5315fae914
15 changed files with 837 additions and 746 deletions
@@ -0,0 +1,279 @@
+# Email Polling System
+
+This document describes the email polling system that replaced the webhook-based approach for real-time email notifications.
+
+## Overview
+
+The email polling system uses **BullMQ** (Redis-based job queue) to periodically check for new emails when users are connected via WebSocket. This approach is more reliable than webhooks since it doesn't depend on external webhook delivery.
+
+## Architecture
+
+### Components
+
+1. **EmailPollingService** - Manages polling jobs for connected users
+2. **EmailPollingProcessor** - Processes email polling jobs and checks for new emails
+3. **EmailGateway** - WebSocket gateway that starts/stops polling on user connection/disconnection
+4. **BullMQ Queue** - Redis-based job queue for scheduling recurring email checks
+
+### Flow
+
+```
+User connects to WebSocket
+    ↓
+EmailGateway.handleConnection()
+    ↓
+EmailPollingService.startPollingForUser()
+    ↓
+BullMQ creates recurring job (every 30 seconds)
+    ↓
+EmailPollingProcessor.process() - checks for new emails
+    ↓
+If new emails found → notify via WebSocket
+    ↓
+User disconnects
+    ↓
+EmailGateway.handleDisconnect()
+    ↓
+EmailPollingService.stopPollingForUser()
+```
+
+## Configuration
+
+### Environment Variables
+
+Required environment variables (see `environment.sample`):
+
+```bash
+# Redis for BullMQ
+REDIS_URI=redis://localhost:6379
+
+# WildDuck Mail Server
+WILDDUCK_SERVER=http://localhost:8080
+WILDDUCK_API_TOKEN=your-wildduck-access-token
+
+# Email Polling (optional)
+EMAIL_POLLING_INTERVAL=30000  # 30 seconds
+EMAIL_POLLING_ENABLED=true
+```
+
+### BullMQ Configuration
+
+The system uses the existing BullMQ configuration from `src/configs/bullmq.config.ts`:
+
+```typescript
+export function bullMqConfig(): SharedBullAsyncConfiguration {
+  return {
+    connection: {
+      url: configService.getOrThrow<string>("REDIS_URI"),
+    },
+    prefix: "dmail",
+    defaultJobOptions: {
+      removeOnComplete: true,
+      removeOnFail: false,
+      attempts: 5,
+    },
+  };
+}
+```
+
+## Implementation Details
+
+### Email Polling Service
+
+Located at `src/modules/email-utils/services/email-polling.service.ts`
+
+**Key Methods:**
+
+- `startPollingForUser(userId, connectionId)` - Start polling when user connects
+- `stopPollingForUser(userId)` - Stop polling when user disconnects
+- `getPollingStats()` - Get statistics about active polling jobs
+
+**Job Configuration:**
+
+```typescript
+const job = await this.emailPollingQueue.add(`email-poll-${userId}`, jobData, {
+  repeat: {
+    every: 30000, // 30 seconds interval
+  },
+  removeOnComplete: 5,
+  removeOnFail: 10,
+  attempts: 3,
+  backoff: {
+    type: "exponential",
+    delay: 2000,
+  },
+});
+```
+
+### Email Polling Processor
+
+Located at `src/modules/email-utils/queue/email-polling.processor.ts`
+
+**Processing Logic:**
+
+1. Check if user is still connected (skip if disconnected)
+2. Get user's mailbox information
+3. Query WildDuck API for recent emails since last check
+4. For each new email:
+   - Send WebSocket notification via `EmailGateway.notifyNewEmail()`
+   - Update unread count via `EmailGateway.notifyUnreadCountUpdate()`
+5. Update job data with new timestamp for next run
+
+### WebSocket Integration
+
+The `EmailGateway` automatically manages polling:
+
+```typescript
+// On user connection
+await this.emailPollingService.startPollingForUser(user.id, client.id);
+
+// On user disconnection
+await this.emailPollingService.stopPollingForUser(userInfo.userId);
+```
+
+## WebSocket Events
+
+The system emits these WebSocket events when new emails are found:
+
+### `new_email` Event
+
+```typescript
+{
+  userId: string;
+  messageId: number;
+  subject: string;
+  from: { name?: string; address: string };
+  to: Array<{ name?: string; address: string }>;
+  hasAttachments: boolean;
+  timestamp: string;
+  mailboxId: string;
+  mailboxName: string;
+  isRead: boolean;
+}
+```
+
+### `unread_count_updated` Event
+
+```typescript
+{
+  userId: string;
+  totalUnread: number;
+  mailboxCounts: Record<string, number>;
+  count: number;
+  timestamp: string;
+}
+```
+
+## Monitoring & Debugging
+
+### Check Active Polling Jobs
+
+```typescript
+const stats = emailPollingService.getPollingStats();
+console.log({
+  activeJobs: stats.activeJobs,
+  userIds: stats.userIds,
+  connections: stats.connections,
+});
+```
+
+### BullMQ Dashboard
+
+You can use BullMQ's web dashboard to monitor jobs:
+
+```bash
+npm install -g @bull-board/express
+# Then access the dashboard at http://localhost:3000/admin/queues
+```
+
+### Logs
+
+The system logs important events:
+
+```
+[EmailPollingService] Started email polling for user 12345 with job ID: email-poll-12345
+[EmailPollingProcessor] Found 3 new emails for user 12345
+[EmailPollingProcessor] Notified user 12345 of 3 new emails
+[EmailPollingService] Stopped email polling for user 12345
+```
+
+## Performance Considerations
+
+### Polling Interval
+
+- Default: 30 seconds
+- Adjustable via `EMAIL_POLLING_INTERVAL` environment variable
+- Balance between real-time updates and API load
+
+### Job Management
+
+- Jobs are automatically cleaned up when users disconnect
+- Failed jobs are retried up to 3 times with exponential backoff
+- Completed jobs are kept (last 5) for debugging
+
+### Resource Usage
+
+- Each connected user creates one recurring BullMQ job
+- Jobs only run when users are connected (checked at start of each job)
+- Redis is used for job queue storage
+
+## Advantages over Webhooks
+
+1. **Reliability** - No dependency on external webhook delivery
+2. **Connection Awareness** - Only polls when users are actually connected
+3. **Error Handling** - Built-in retry mechanism with exponential backoff
+4. **Monitoring** - Full visibility into job status and performance
+5. **Scalability** - Redis-based queue can handle high loads
+6. **Control** - Can adjust polling frequency per user if needed
+
+## Migration from Webhooks
+
+The old webhook system has been completely removed:
+
+**Deleted Files:**
+
+- `src/modules/email-utils/controllers/webhook.controller.ts`
+- `src/modules/email-utils/services/webhook-management.service.ts`
+- `src/modules/email-utils/services/webhook-email-events.service.ts`
+- `src/modules/email/interfaces/webhook-events.interface.ts`
+
+**Updated Files:**
+
+- `src/modules/email-utils/email-utils.module.ts` - Added BullMQ and new services
+- `src/modules/email-gateway/email.gateway.ts` - Added polling integration
+- `src/modules/email-gateway/email-gateway.module.ts` - Added EmailUtilsModule import
+
+## Troubleshooting
+
+### Common Issues
+
+**Issue: Jobs not running**
+
+- Check Redis connection
+- Verify `REDIS_URI` environment variable
+- Check BullMQ logs
+
+**Issue: Users not receiving notifications**
+
+- Verify user is connected to WebSocket
+- Check if polling job is active: `emailPollingService.getPollingStats()`
+- Verify WildDuck API credentials and server URL
+
+**Issue: High Redis memory usage**
+
+- Adjust `removeOnComplete` and `removeOnFail` values
+- Consider shorter polling intervals for fewer concurrent jobs
+
+**Issue: API rate limiting**
+
+- Increase polling interval
+- Implement user-specific rate limiting
+- Use WildDuck API bulk endpoints if available
+
+## Future Enhancements
+
+1. **Dynamic Polling Intervals** - Adjust frequency based on user activity
+2. **Priority Users** - More frequent polling for premium users
+3. **Email Filtering** - Only notify for emails matching user preferences
+4. **Batch Processing** - Process multiple users in single API calls
+5. **Caching** - Cache recent emails to reduce API calls