dmail-api/EMAIL_POLLING_SYSTEM.md

Email Polling System

This document describes the email polling system that replaced the webhook-based approach for real-time email notifications.

Overview

The email polling system uses BullMQ (Redis-based job queue) to periodically check for new emails when users are connected via WebSocket. This approach is more reliable than webhooks since it doesn't depend on external webhook delivery.

Architecture

Components

1. EmailPollingService - Manages polling jobs for connected users
2. EmailPollingProcessor - Processes email polling jobs and checks for new emails
3. EmailGateway - WebSocket gateway that starts/stops polling on user connection/disconnection
4. BullMQ Queue - Redis-based job queue for scheduling recurring email checks

Flow

User connects to WebSocket
    ↓
EmailGateway.handleConnection()
    ↓
EmailPollingService.startPollingForUser()
    ↓
BullMQ creates recurring job (every 30 seconds)
    ↓
EmailPollingProcessor.process() - checks for new emails
    ↓
If new emails found → notify via WebSocket
    ↓
User disconnects
    ↓
EmailGateway.handleDisconnect()
    ↓
EmailPollingService.stopPollingForUser()

Configuration

Environment Variables

Required environment variables (see environment.sample):

# Redis for BullMQ
REDIS_URI=redis://localhost:6379

# WildDuck Mail Server
WILDDUCK_SERVER=http://localhost:8080
WILDDUCK_API_TOKEN=your-wildduck-access-token

# Email Polling (optional)
EMAIL_POLLING_INTERVAL=30000  # 30 seconds
EMAIL_POLLING_ENABLED=true

BullMQ Configuration

The system uses the existing BullMQ configuration from src/configs/bullmq.config.ts:

export function bullMqConfig(): SharedBullAsyncConfiguration {
  return {
    connection: {
      url: configService.getOrThrow<string>("REDIS_URI"),
    },
    prefix: "dmail",
    defaultJobOptions: {
      removeOnComplete: true,
      removeOnFail: false,
      attempts: 5,
    },
  };
}

Implementation Details

Email Polling Service

Located at src/modules/email-utils/services/email-polling.service.ts

Key Methods:

• startPollingForUser(userId, connectionId) - Start polling when user connects
• stopPollingForUser(userId) - Stop polling when user disconnects
• getPollingStats() - Get statistics about active polling jobs

Job Configuration:

const job = await this.emailPollingQueue.add(`email-poll-${userId}`, jobData, {
  repeat: {
    every: 30000, // 30 seconds interval
  },
  removeOnComplete: 5,
  removeOnFail: 10,
  attempts: 3,
  backoff: {
    type: "exponential",
    delay: 2000,
  },
});

Email Polling Processor

Located at src/modules/email-utils/queue/email-polling.processor.ts

Processing Logic:

1. Check if user is still connected (skip if disconnected)
2. Get user's mailbox information
3. Query WildDuck API for recent emails since last check
4. For each new email:
   • Send WebSocket notification via EmailGateway.notifyNewEmail()
   • Update unread count via EmailGateway.notifyUnreadCountUpdate()
5. Update job data with new timestamp for next run

WebSocket Integration

The EmailGateway automatically manages polling:

// On user connection
await this.emailPollingService.startPollingForUser(user.id, client.id);

// On user disconnection
await this.emailPollingService.stopPollingForUser(userInfo.userId);

WebSocket Events

The system emits these WebSocket events when new emails are found:

new_email Event

{
  userId: string;
  messageId: number;
  subject: string;
  from: { name?: string; address: string };
  to: Array<{ name?: string; address: string }>;
  hasAttachments: boolean;
  timestamp: string;
  mailboxId: string;
  mailboxName: string;
  isRead: boolean;
}

unread_count_updated Event

{
  userId: string;
  totalUnread: number;
  mailboxCounts: Record<string, number>;
  count: number;
  timestamp: string;
}

Monitoring & Debugging

Check Active Polling Jobs

const stats = emailPollingService.getPollingStats();
console.log({
  activeJobs: stats.activeJobs,
  userIds: stats.userIds,
  connections: stats.connections,
});

BullMQ Dashboard

You can use BullMQ's web dashboard to monitor jobs:

npm install -g @bull-board/express
# Then access the dashboard at http://localhost:3000/admin/queues

Logs

The system logs important events:

[EmailPollingService] Started email polling for user 12345 with job ID: email-poll-12345
[EmailPollingProcessor] Found 3 new emails for user 12345
[EmailPollingProcessor] Notified user 12345 of 3 new emails
[EmailPollingService] Stopped email polling for user 12345

Performance Considerations

Polling Interval

• Default: 30 seconds
• Adjustable via EMAIL_POLLING_INTERVAL environment variable
• Balance between real-time updates and API load

Job Management

• Jobs are automatically cleaned up when users disconnect
• Failed jobs are retried up to 3 times with exponential backoff
• Completed jobs are kept (last 5) for debugging

Resource Usage

• Each connected user creates one recurring BullMQ job
• Jobs only run when users are connected (checked at start of each job)
• Redis is used for job queue storage

Advantages over Webhooks

1. Reliability - No dependency on external webhook delivery
2. Connection Awareness - Only polls when users are actually connected
3. Error Handling - Built-in retry mechanism with exponential backoff
4. Monitoring - Full visibility into job status and performance
5. Scalability - Redis-based queue can handle high loads
6. Control - Can adjust polling frequency per user if needed

Migration from Webhooks

The old webhook system has been completely removed:

Deleted Files:

• src/modules/email-utils/controllers/webhook.controller.ts
• src/modules/email-utils/services/webhook-management.service.ts
• src/modules/email-utils/services/webhook-email-events.service.ts
• src/modules/email/interfaces/webhook-events.interface.ts

Updated Files:

• src/modules/email-utils/email-utils.module.ts - Added BullMQ and new services
• src/modules/email-gateway/email.gateway.ts - Added polling integration
• src/modules/email-gateway/email-gateway.module.ts - Added EmailUtilsModule import

Troubleshooting

Common Issues

Issue: Jobs not running

• Check Redis connection
• Verify REDIS_URI environment variable
• Check BullMQ logs

Issue: Users not receiving notifications

• Verify user is connected to WebSocket
• Check if polling job is active: emailPollingService.getPollingStats()
• Verify WildDuck API credentials and server URL

Issue: High Redis memory usage

• Adjust removeOnComplete and removeOnFail values
• Consider shorter polling intervals for fewer concurrent jobs

Issue: API rate limiting

• Increase polling interval
• Implement user-specific rate limiting
• Use WildDuck API bulk endpoints if available

Future Enhancements

1. Dynamic Polling Intervals - Adjust frequency based on user activity
2. Priority Users - More frequent polling for premium users
3. Email Filtering - Only notify for emails matching user preferences
4. Batch Processing - Process multiple users in single API calls
5. Caching - Cache recent emails to reduce API calls