Evolution of Request Routing & Handling: From CGI to Modern Web

April 8, 2025 by Dan

The way web servers route and handle requests has evolved dramatically over the past three decades. This evolution reflects not just technological advancements, but also changing needs for performance, security, and developer experience. Let's explore this journey from the earliest days of CGI scripts to today's sophisticated routing frameworks.

Early CGI Era (1993-1997)

The Common Gateway Interface (CGI) was the first standardized way to generate dynamic content on the web:

1:1 File-to-URL Mapping: Each script had its own dedicated URL path
Complete Process Spawning: Every request launched a new process
Query String Parameters: Data passed primarily through URL parameters
Limited HTTP Method Support: Primarily GET and POST
Simple Directory Structure: Scripts commonly placed in /cgi-bin/ directory
Performance Limitations: New process for each request created high overhead

# Typical Perl CGI script (circa 1994)
#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "<html><body>";

# Parse query string manually
if ($ENV{'QUERY_STRING'} =~ /name=([^&]+)/) {
    $name = $1;
    print "<h1>Hello, $name</h1>";
} else {
    print "<h1>Hello, world</h1>";
}

print "</body></html>";

Server configuration was minimal, focused primarily on designating which directories could execute scripts:

# NCSA httpd.conf (circa 1995)
ScriptAlias /cgi-bin/ /usr/local/www/cgi-bin/

<Directory /usr/local/www/cgi-bin>
    Options ExecCGI
    AllowOverride None
</Directory>

This approach was straightforward but inefficient, as each request required spawning a new process. To handle high traffic, websites often needed to distribute load across multiple servers, frequently using DNS round-robin with hosts like www1, www2, www3, etc.

Server Extensions & Modules (1996-2000)

To address CGI's performance limitations, server-specific extensions emerged:

Apache Modules: mod_perl, mod_php enabling in-process execution
Microsoft IIS: ISAPI Extensions via DLLs for Windows servers
Netscape Server: NSAPI for custom server extensions
FastCGI: Long-running processes handling multiple requests
.htaccess: Apache's per-directory configuration for URL rewriting
Java Servlets: Early standardized server-side component technology

# Apache mod_rewrite example (.htaccess circa 1998)
RewriteEngine On
RewriteBase /
RewriteRule ^product/([0-9]+)$ product.php?id=$1 [L]
RewriteRule ^category/([a-z]+)$ category.php?name=$1 [L]

Java Servlets introduced a more structured approach to request handling:

// Java Servlet (circa 1998)
public class ProductServlet extends HttpServlet {
    public void doGet(HttpServletRequest request, 
                     HttpServletResponse response) 
                     throws ServletException, IOException {
        
        String productId = request.getParameter("id");
        response.setContentType("text/html");
        PrintWriter out = response.getWriter();
        
        out.println("<html><body>");
        out.println("<h1>Product ID: " + productId + "</h1>");
        out.println("</body></html>");
    }
}

Custom DLL files in IIS allowed Windows developers to create server extensions:

// ISAPI Extension (circa 1997)
BOOL WINAPI GetExtensionVersion(HSE_VERSION_INFO* pVer) {
    pVer->dwExtensionVersion = MAKELONG(HSE_VERSION_MINOR, HSE_VERSION_MAJOR);
    strncpy(pVer->lpszExtensionDesc, "Product Handler", HSE_MAX_EXT_DLL_NAME_LEN);
    return TRUE;
}

DWORD WINAPI HttpExtensionProc(LPEXTENSION_CONTROL_BLOCK lpEcb) {
    char buffer[1024];
    sprintf(buffer, "Content-type: text/html\r\n\r\n"
                   "<html><body>"
                   "<h1>Product ID: %s</h1>"
                   "</body></html>", 
            lpEcb->GetServerVariable(lpEcb->ConnID, "QUERY_STRING"));
    
    lpEcb->WriteClient(lpEcb->ConnID, buffer, strlen(buffer), 0);
    return HSE_STATUS_SUCCESS;
}

This era saw the beginning of more sophisticated request routing, but still relied heavily on filename-based mapping with simple URL rewriting rules.

HTTP Protocol Evolution (1996-2000)

As the web grew, the HTTP protocol itself evolved to address emerging needs:

HTTP/1.0 (1996): Basic protocol with limited headers
HTTP/1.1 (1997):
- Added Host header allowing virtual hosting (multiple sites on one IP)
- Persistent connections reducing TCP overhead
- Chunked transfer encoding for streaming
Compression: Content-Encoding and Accept-Encoding headers for gzip
Caching Controls: ETag, If-Modified-Since for improved caching
Cookies (1997): State management via Set-Cookie headers

# HTTP/1.1 Request with Host header (1997)
GET /product/123 HTTP/1.1
Host: www.example.com
Accept: text/html
Accept-Encoding: gzip
Cookie: session=abc123

# HTTP/1.1 Response with compression
HTTP/1.1 200 OK
Content-Type: text/html
Content-Encoding: gzip
Set-Cookie: session=abc123; path=/
Content-Length: 438

[compressed content]

The Host header was particularly revolutionary, as it solved the IP address shortage problem by allowing multiple websites to share a single IP address. This became known as "virtual hosting" and fundamentally changed how web servers routed requests.

Direct File Mapping Era (1998-2004)

PHP popularized a direct file-to-URL mapping approach that streamlined development:

PHP Files: URLs mapped directly to .php files in directories
ASP Pages: Similar approach with .asp files on Windows servers
JSP: Java Server Pages following the same pattern for Java
URL Parameters: Both querystring and increasingly POST data
Shared Hosting: Simplified deployment model for small sites
Convention Over Configuration: Default routing based on filesystem

# PHP file (circa 2000) with direct file-to-URL mapping
<?php
// File: /product.php - Accessed via /product.php?id=123
$product_id = $_GET['id'];

// Database connection
$conn = mysql_connect("localhost", "user", "password");
mysql_select_db("products", $conn);

// Fetch product
$result = mysql_query("SELECT * FROM products WHERE id = $product_id");
$product = mysql_fetch_assoc($result);
?>

<html>
<head><title>Product: <?php echo $product['name']; ?></title></head>
<body>
    <h1><?php echo $product['name']; ?></h1>
    <p>Price: $<?php echo $product['price']; ?></p>
    <p><?php echo $product['description']; ?></p>
</body>
</html>

This direct mapping approach was easy to understand and deploy, particularly in shared hosting environments that dominated the early 2000s. However, it often led to security vulnerabilities (SQL injection in the example above) and maintenance challenges as applications grew larger.

Front Controller Pattern (2002-2008)

As applications grew more complex, the "Front Controller" pattern emerged to centralize request handling:

Single Entry Point: All requests routed through index.php
URL Rewriting: Apache/Nginx rules to support clean URLs
Pattern Matching: Rules for mapping URLs to controllers
MVC Frameworks: Structured approach to routing in Rails, Struts, etc.
Separation of Concerns: Routing separate from business logic
Route Parameters: Dynamic segments in URL paths

# Apache rewrite rules for front controller (circa 2005)
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    
    # Exclude direct access to actual files and directories
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    
    # Route everything else to index.php
    RewriteRule ^(.*)$ index.php?path=$1 [QSA,L]
</IfModule>

# PHP front controller (circa 2005)
<?php
// index.php - All requests come here
$path = $_GET['path'] ?? '';
$segments = explode('/', trim($path, '/'));

// Determine controller and action
$controller_name = !empty($segments[0]) ? $segments[0] : 'home';
$action_name = !empty($segments[1]) ? $segments[1] : 'index';

// Extract parameters
$params = array_slice($segments, 2);

// Load and execute controller
$controller_file = "controllers/{$controller_name}_controller.php";
if (file_exists($controller_file)) {
    include_once($controller_file);
    $controller_class = ucfirst($controller_name) . 'Controller';
    $controller = new $controller_class();
    
    if (method_exists($controller, $action_name)) {
        call_user_func_array([$controller, $action_name], $params);
    } else {
        show_404();
    }
} else {
    show_404();
}
?>

Ruby on Rails (2005) popularized convention-based routing with its "RESTful" approach:

# Ruby on Rails routes (circa 2006)
ActionController::Routing::Routes.draw do |map|
  map.resources :products
  map.resources :categories do |categories|
    categories.resources :products
  end
  
  map.root :controller => "home"
end

This era introduced a more structured approach to URL mapping, with centralized control and separation of routing from application logic.

Load Balancing & Reverse Proxies (2000-2010)

As web applications needed to scale beyond single servers, routing became more complex at the infrastructure level:

Hardware Load Balancers: F5, Cisco, etc. for high-traffic sites
Software Load Balancers: HAProxy, Nginx, Apache mod_proxy
DNS Round Robin: Multiple A records (www1, www2, etc.) for simple distribution
Session Affinity: "Sticky sessions" to keep users on the same server
Reverse Proxies: Terminating connections and routing to backend services
Health Checks: Dynamic routing based on server availability

# Nginx as reverse proxy and load balancer (circa 2008)
upstream web_servers {
    ip_hash;  # Session affinity
    server web1.example.com:8080;
    server web2.example.com:8080;
    server web3.example.com:8080 backup;
}

server {
    listen 80;
    server_name www.example.com;
    
    location / {
        proxy_pass http://web_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
    
    location /static/ {
        root /var/www/static;
        expires 30d;
    }
}

Load balancing decisions evolved from simple round-robin to more sophisticated algorithms:

Least Connections: Route to server with fewest active connections
Response Time: Route based on server performance
Geographic: Route to servers closest to the user
Content-Based: Different routing for static vs. dynamic content

HTTPS & Certificate Evolution (1995-Present)

Security became an increasingly important aspect of request routing:

SSL/TLS Termination: Processing encryption at load balancers
Certificate Evolution:
- 1995-2010: Expensive certificates from limited authorities
- 2010-2015: Cheaper alternatives emerge
- 2015-Present: Let's Encrypt providing free, automated certificates
SNI (Server Name Indication): Multiple SSL certificates on one IP
Certificate Transparency: Public logging of all certificates
HTTP to HTTPS Redirects: Automatic routing to secure version
HSTS: Browser-enforced secure connections

# Modern Nginx HTTPS configuration (circa 2020)
server {
    listen 80;
    server_name example.com www.example.com;
    
    # Redirect all HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name example.com www.example.com;
    
    # Let's Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    
    # Strong SSL configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:...;
    
    # HSTS (15768000 seconds = 6 months)
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains" always;
    
    # Rest of configuration...
}

The path to ubiquitous HTTPS has been transformative for web security. With Let's Encrypt removing financial barriers, secure connections have become the default rather than the exception.

Media Streaming Evolution (1995-Present)

Streaming media required specialized routing solutions:

Early Solutions:
- Server push (multipart/x-mixed-replace) for Motion JPEG
- Progressive download of media files
- RealPlayer, Windows Media, QuickTime streaming protocols
Flash Era: RTMP (Real-Time Messaging Protocol) dominance
Modern Standards:
- HTTP Live Streaming (HLS) for Apple devices
- MPEG-DASH for standardized streaming
- WebRTC for peer-to-peer streaming
CDN Integration: Specialized edge routing for media delivery
Adaptive Bitrate: Dynamic quality selection based on conditions

# Nginx RTMP configuration (circa 2015)
rtmp {
    server {
        listen 1935;
        
        application live {
            live on;
            record off;
            
            # HLS
            hls on;
            hls_path /var/www/html/hls;
            hls_fragment 3;
            hls_playlist_length 60;
            
            # DASH
            dash on;
            dash_path /var/www/html/dash;
        }
    }
}

Modern streaming solutions have largely converged on HTTP-based protocols, allowing them to leverage standard web infrastructure and caching mechanisms. This represents a significant shift from the proprietary protocols of the early 2000s.

Declarative Framework Routing (2010-Present)

Modern frameworks offer sophisticated declarative routing mechanisms:

Annotation/Decorator Routing: Metadata directly in controller code
Lambda/Middleware Pipelines: Functional approach to request processing
Content Negotiation: Automatic format selection (JSON, XML, HTML)
Parameter Validation: Type checking and constraints at the routing level
Versioned APIs: Routing to different implementations by version
Feature Flags: Routing based on gradual rollout controls

# Express.js routing (Node.js, circa 2015)
import express from 'express';
const app = express();

// Middleware pipeline
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.use(cookieParser());

// Parameter validation middleware
const validateProduct = (req, res, next) => {
  const { id } = req.params;
  if (!id || !/^\d+$/.test(id)) {
    return res.status(400).json({ error: 'Invalid product ID' });
  }
  next();
};

// Routes with middleware
app.get('/api/products', async (req, res) => {
  const products = await ProductService.getAll();
  res.json(products);
});

app.get('/api/products/:id', validateProduct, async (req, res) => {
  const product = await ProductService.getById(req.params.id);
  if (!product) {
    return res.status(404).json({ error: 'Product not found' });
  }
  res.json(product);
});

// Content negotiation
app.get('/products/:id', validateProduct, async (req, res) => {
  const product = await ProductService.getById(req.params.id);
  
  // Respond based on Accept header
  res.format({
    'application/json': () => res.json(product),
    'text/html': () => res.render('product', { product }),
    'default': () => res.status(406).send('Not Acceptable')
  });
});

Spring Boot demonstrates annotation-based routing:

# Spring Boot controller (circa 2018)
@RestController
@RequestMapping("/api/products")
public class ProductController {
    
    private final ProductService productService;
    
    public ProductController(ProductService productService) {
        this.productService = productService;
    }
    
    @GetMapping
    public List getAllProducts() {
        return productService.findAll();
    }
    
    @GetMapping("/{id}")
    public ResponseEntity getProduct(@PathVariable Long id) {
        return productService.findById(id)
                .map(ResponseEntity::ok)
                .orElse(ResponseEntity.notFound().build());
    }
    
    @PostMapping
    @ResponseStatus(HttpStatus.CREATED)
    public Product createProduct(@Valid @RequestBody ProductDTO productDTO) {
        return productService.create(productDTO);
    }
    
    @PutMapping("/{id}")
    public ResponseEntity updateProduct(
            @PathVariable Long id, 
            @Valid @RequestBody ProductDTO productDTO) {
        return productService.update(id, productDTO)
                .map(ResponseEntity::ok)
                .orElse(ResponseEntity.notFound().build());
    }
    
    @DeleteMapping("/{id}")
    @ResponseStatus(HttpStatus.NO_CONTENT)
    public void deleteProduct(@PathVariable Long id) {
        productService.delete(id);
    }
}

Modern frameworks have built sophisticated request processing pipelines that handle routing, parameter validation, content negotiation, and more in a declarative, convention-based manner.

API Gateways & Microservices (2015-Present)

In the microservices era, routing has expanded to service-level concerns:

API Gateways: Centralized entry points for all services
Service Discovery: Dynamic routing to service instances
Circuit Breaking: Routing around failing services
Rate Limiting: Throttling requests at the gateway level
Request Transformation: Modifying requests between services
Authentication/Authorization: Gateway-level security
Analytics & Monitoring: Tracking API usage patterns

# Kong API Gateway configuration (circa 2020)
services:
  - name: product-service
    url: http://product-service:8080
    routes:
      - name: product-routes
        paths:
          - /api/products
          - /api/categories
        strip_path: false
    plugins:
      - name: rate-limiting
        config:
          minute: 60
          policy: local
      - name: jwt
        config:
          secret_is_base64: false
          claims_to_verify:
            - exp
  
  - name: order-service
    url: http://order-service:8080
    routes:
      - name: order-routes
        paths:
          - /api/orders
        strip_path: false
    plugins:
      - name: cors
      - name: request-transformer
        config:
          add:
            headers:
              - X-Consumer-ID:$(consumer.id)

Modern service meshes provide even more sophisticated routing capabilities:

# Istio routing rules (circa 2022)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: product-service
spec:
  hosts:
    - product-service
  http:
    - match:
        - headers:
            x-api-version:
              exact: "2.0"
      route:
        - destination:
            host: product-service-v2
            port:
              number: 8080
          weight: 90
        - destination:
            host: product-service-v2-beta
            port:
              number: 8080
          weight: 10
    - route:
        - destination:
            host: product-service-v1
            port:
              number: 8080

This level of routing intelligence enables advanced patterns like canary deployments, A/B testing, and targeted feature rollouts, all managed at the infrastructure layer rather than within application code.

HTTP Protocol Continued Evolution

The HTTP protocol itself continues to evolve, affecting routing and request handling:

HTTP/2 (2015):
- Multiplexing multiple requests over one connection
- Header compression reducing overhead
- Server push for proactive resource delivery
HTTP/3 (2020-Present):
- QUIC protocol replacing TCP with UDP-based transport
- Improved connection migration between networks
- Reduced handshake latency
WebSockets: Bi-directional communication channel
Server-Sent Events: One-way server-to-client streaming

These protocol enhancements dramatically change the performance characteristics of web requests, though they're largely transparent to application developers.

Edge Computing & CDNs (2018-Present)

The newest frontier in request routing is moving computation closer to users:

Edge Functions: Cloudflare Workers, Lambda@Edge, Vercel Edge
Edge-First Frameworks: Next.js, SvelteKit with edge rendering
Geolocation Routing: Serving different content based on user location
Edge Databases: Low-latency globally distributed data
Feature Flagging: Controlling features at the edge layer
A/B Testing: Traffic splitting for experiments

# Cloudflare Worker routing (circa 2022)
// Edge function deployed globally
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url)
  
  // Routing at the edge
  if (url.pathname.startsWith('/api/')) {
    return handleApiRequest(request)
  }
  
  // A/B testing at the edge
  if (url.pathname === '/') {
    // Randomly assign to variant A or B
    const variant = Math.random() < 0.5 ? 'A' : 'B'
    
    // Get appropriate content
    const response = await getVariant(variant)
    
    // Set cookie for consistency
    response.headers.set('Set-Cookie', `variant=${variant}; path=/`)
    
    return response
  }
  
  // Geolocation-based routing
  const country = request.headers.get('CF-IPCountry')
  if (country && url.pathname === '/pricing') {
    return getPricingForCountry(country)
  }
  
  // Default - pass to origin
  return fetch(request)
}

Edge computing represents a significant shift in the routing paradigm, moving logic traditionally handled at the application server level to a distributed network of edge nodes running closer to end users.

The Future of Request Routing

Looking ahead, several trends are likely to shape the future of request routing:

AI-Enhanced Routing: Using machine learning to predict optimal routes
Intent-Based Routing: Routing based on inferred user intent rather than explicit paths
Privacy-Preserving Proxies: Intermediaries that strip identifying information
Decentralized Routing: Peer-to-peer and blockchain-based alternatives to centralized DNS
Ambient Computing: Seamless routing across devices and environments

The evolution of request routing reflects a continuous push toward better performance, security, developer experience, and user outcomes. From the simple mapping of URLs to files in the CGI era to the sophisticated, distributed intelligence of modern edge computing, each advancement has enabled new capabilities while addressing the limitations of previous approaches.

Evolution of Response Generation - Discover how response generation techniques evolved alongside request handling
Comprehensive List of Web Framework Responsibilities - See how request routing fits into the broader web framework ecosystem
Evolution of Session & State Management - Explore how state management evolved alongside request handling
The Evolution of Internet Clients - Understand how client devices shaped request handling approaches

Evolution of Request Routing & Handling: From CGI to Modern Web

The Future of Request Routing

Related Articles

About

Archives

Optimize Your Website!

Elsewhere