dmail-api/EMAIL_POLLING_SYSTEM.md

# Email Polling System

This document describes the email polling system that replaced the webhook-based approach for real-time email notifications.

## Overview

The email polling system uses **BullMQ** (Redis-based job queue) to periodically check for new emails when users are connected via WebSocket. This approach is more reliable than webhooks since it doesn't depend on external webhook delivery.

## Architecture

### Components

1. **EmailPollingService** - Manages polling jobs for connected users
2. **EmailPollingProcessor** - Processes email polling jobs and checks for new emails
3. **EmailGateway** - WebSocket gateway that starts/stops polling on user connection/disconnection
4. **BullMQ Queue** - Redis-based job queue for scheduling recurring email checks

### Flow

```
User connects to WebSocket
    ↓
EmailGateway.handleConnection()
    ↓
EmailPollingService.startPollingForUser()
    ↓
BullMQ creates recurring job (every 30 seconds)
    ↓
EmailPollingProcessor.process() - checks for new emails
    ↓
If new emails found → notify via WebSocket
    ↓
User disconnects
    ↓
EmailGateway.handleDisconnect()
    ↓
EmailPollingService.stopPollingForUser()
```

## Configuration

### Environment Variables

Required environment variables (see `environment.sample`):

```bash
# Redis for BullMQ
REDIS_URI=redis://localhost:6379

# WildDuck Mail Server
WILDDUCK_SERVER=http://localhost:8080
WILDDUCK_API_TOKEN=your-wildduck-access-token

# Email Polling (optional)
EMAIL_POLLING_INTERVAL=30000  # 30 seconds
EMAIL_POLLING_ENABLED=true
```

### BullMQ Configuration

The system uses the existing BullMQ configuration from `src/configs/bullmq.config.ts`:

```typescript
export function bullMqConfig(): SharedBullAsyncConfiguration {
  return {
    connection: {
      url: configService.getOrThrow<string>("REDIS_URI"),
    },
    prefix: "dmail",
    defaultJobOptions: {
      removeOnComplete: true,
      removeOnFail: false,
      attempts: 5,
    },
  };
}
```

## Implementation Details

### Email Polling Service

Located at `src/modules/email-utils/services/email-polling.service.ts`

**Key Methods:**

- `startPollingForUser(userId, connectionId)` - Start polling when user connects
- `stopPollingForUser(userId)` - Stop polling when user disconnects
- `getPollingStats()` - Get statistics about active polling jobs

**Job Configuration:**

```typescript
const job = await this.emailPollingQueue.add(`email-poll-${userId}`, jobData, {
  repeat: {
    every: 30000, // 30 seconds interval
  },
  removeOnComplete: 5,
  removeOnFail: 10,
  attempts: 3,
  backoff: {
    type: "exponential",
    delay: 2000,
  },
});
```

### Email Polling Processor

Located at `src/modules/email-utils/queue/email-polling.processor.ts`

**Processing Logic:**

1. Check if user is still connected (skip if disconnected)
2. Get user's mailbox information
3. Query WildDuck API for recent emails since last check
4. For each new email:
   - Send WebSocket notification via `EmailGateway.notifyNewEmail()`
   - Update unread count via `EmailGateway.notifyUnreadCountUpdate()`
5. Update job data with new timestamp for next run

### WebSocket Integration

The `EmailGateway` automatically manages polling:

```typescript
// On user connection
await this.emailPollingService.startPollingForUser(user.id, client.id);

// On user disconnection
await this.emailPollingService.stopPollingForUser(userInfo.userId);
```

## WebSocket Events

The system emits these WebSocket events when new emails are found:

### `new_email` Event

```typescript
{
  userId: string;
  messageId: number;
  subject: string;
  from: { name?: string; address: string };
  to: Array<{ name?: string; address: string }>;
  hasAttachments: boolean;
  timestamp: string;
  mailboxId: string;
  mailboxName: string;
  isRead: boolean;
}
```

### `unread_count_updated` Event

```typescript
{
  userId: string;
  totalUnread: number;
  mailboxCounts: Record<string, number>;
  count: number;
  timestamp: string;
}
```

## Monitoring & Debugging

### Check Active Polling Jobs

```typescript
const stats = emailPollingService.getPollingStats();
console.log({
  activeJobs: stats.activeJobs,
  userIds: stats.userIds,
  connections: stats.connections,
});
```

### BullMQ Dashboard

You can use BullMQ's web dashboard to monitor jobs:

```bash
npm install -g @bull-board/express
# Then access the dashboard at http://localhost:3000/admin/queues
```

### Logs

The system logs important events:

```
[EmailPollingService] Started email polling for user 12345 with job ID: email-poll-12345
[EmailPollingProcessor] Found 3 new emails for user 12345
[EmailPollingProcessor] Notified user 12345 of 3 new emails
[EmailPollingService] Stopped email polling for user 12345
```

## Performance Considerations

### Polling Interval

- Default: 30 seconds
- Adjustable via `EMAIL_POLLING_INTERVAL` environment variable
- Balance between real-time updates and API load

### Job Management

- Jobs are automatically cleaned up when users disconnect
- Failed jobs are retried up to 3 times with exponential backoff
- Completed jobs are kept (last 5) for debugging

### Resource Usage

- Each connected user creates one recurring BullMQ job
- Jobs only run when users are connected (checked at start of each job)
- Redis is used for job queue storage

## Advantages over Webhooks

1. **Reliability** - No dependency on external webhook delivery
2. **Connection Awareness** - Only polls when users are actually connected
3. **Error Handling** - Built-in retry mechanism with exponential backoff
4. **Monitoring** - Full visibility into job status and performance
5. **Scalability** - Redis-based queue can handle high loads
6. **Control** - Can adjust polling frequency per user if needed

## Migration from Webhooks

The old webhook system has been completely removed:

**Deleted Files:**

- `src/modules/email-utils/controllers/webhook.controller.ts`
- `src/modules/email-utils/services/webhook-management.service.ts`
- `src/modules/email-utils/services/webhook-email-events.service.ts`
- `src/modules/email/interfaces/webhook-events.interface.ts`

**Updated Files:**

- `src/modules/email-utils/email-utils.module.ts` - Added BullMQ and new services
- `src/modules/email-gateway/email.gateway.ts` - Added polling integration
- `src/modules/email-gateway/email-gateway.module.ts` - Added EmailUtilsModule import

## Troubleshooting

### Common Issues

**Issue: Jobs not running**

- Check Redis connection
- Verify `REDIS_URI` environment variable
- Check BullMQ logs

**Issue: Users not receiving notifications**

- Verify user is connected to WebSocket
- Check if polling job is active: `emailPollingService.getPollingStats()`
- Verify WildDuck API credentials and server URL

**Issue: High Redis memory usage**

- Adjust `removeOnComplete` and `removeOnFail` values
- Consider shorter polling intervals for fewer concurrent jobs

**Issue: API rate limiting**

- Increase polling interval
- Implement user-specific rate limiting
- Use WildDuck API bulk endpoints if available

## Future Enhancements

1. **Dynamic Polling Intervals** - Adjust frequency based on user activity
2. **Priority Users** - More frequent polling for premium users
3. **Email Filtering** - Only notify for emails matching user preferences
4. **Batch Processing** - Process multiple users in single API calls
5. **Caching** - Cache recent emails to reduce API calls