How It Works

How It Works

The engineering behind the fast dashboard experience

Data Flow

Police APIRaw data source
Cron JobBatch processing
MongoDBPre-computed stats
ClientRead-only dashboard

Performance Metrics

Records Stored
~500

Pre-computed aggregations instead of 100k+ raw records

Average Load Time
48ms

300x faster than computing from raw data (15+ seconds)

Data Coverage
28 months

Complete Metropolitan Police stop and search history

The Pre-computation Insight

The Problem

The Metropolitan Police API provides individual stop and search records. Storing and querying 100,000+ raw records in the browser leads to:

  • 15+ second load times
  • Slow client-side aggregations
  • Poor user experience
  • Wasted API calls and bandwidth
The Solution

Instead of storing raw records, pre-compute all possible filter combinations:

28 months
× 6 age ranges (including null)
× 3 search types
= 504 combinations

Each combination stores aggregated statistics (counts, percentages, breakdowns). This reduces the dataset from 100k+ records to ~500, enabling instant filtering and visualization.

Why Separate Cron Job?

The data ingestion process is handled by a separate cron job rather than the client:

  • Rate limit safety - Batch processing with controlled delays prevents API throttling
  • No race conditions - Single source of truth for data updates
  • Separation of concerns - Client is purely read-only, cron handles writes
  • Better UX - Users never wait for expensive ingestion operations
  • Simpler debugging - Clear boundary between data fetch and display logic
The Result

A 300x performance improvement that makes the dashboard feel instant:

  • 48ms average load time vs 15+ seconds
  • All filter combinations pre-computed and ready
  • Client remains simple and maintainable
  • No impact on API rate limits during user sessions