HAProxy Timeouts: How To Fix 504 Gateway Timeout Error

Overview:

This guide explains the various timeout parameters in HAProxy and how to fix the 504 gateway timeout error in HAProxy.

Foreword: Meaning of 504 Gateway Timeout Error

The HTTP 504 Gateway Timeout error is a common server server-side error or status code faced by users and website owners. This error indicates that while acting as a gateway or proxy, a server (HAProxy in the case of this guide) did not get a timely response from the upstream server that it is proxying requests to, to complete the request.

To fix this error in HAProxy, you must understand some of the common HAProxy timeout parameters and know which one to adjust accordingly.

Key HAProxy Timeout Parameters

The following are the key and default HAProxy timeout settings that you need to adjust in your configuration file. By default, the time is assumed to be in milliseconds. You can use ‘s’ or ‘m’ or ‘h’ to denote seconds, minutes, or hours respectively:

• timeout connect: this parameter defines the time that HAProxy will wait for a TCP connection to a backend server to be established. This setting can only be added in the default, listen, and backend sections.
• timeout client: this parameter defines the maximum inactivity time on the client side. This setting may be used in the default, listen, frontend, and backend sections.
• timeout server: this setting defines the maximum inactivity time on the upstream server side. Once the timeout expires, the connection to the backend is closed. This setting can only be added in the default, listen, and backend sections.

Note: If you have deployed HAProxy in TCP mode, the timeout server parameter value should be the same as the timeout client parameter value. Having different values makes confusion more likely because HAProxy doesn’t know which side is supposed to be speaking and, since both apply all the time.

Also read: When To Run HAProxy in HTTP or TCP Modes

Here are additional timeout settings that you can also take advantage of:

• timeout client-fin: this setting defines the inactivity timeout on the client side for half-closed connections. This setting may only be used in the default, listen, and backend sections.
• timeout server-fin: this setting defines the inactivity timeout on the server side for half-closed connections. This setting can only be added in the default, listen, and backend sections.
• timeout queue: this setting defines the maximum time to wait in the queue for a connection slot to be free after the maxconn is reached.
• timeout check: this parameter is used to set additional check timeout, but only after a connection has been already established. It is applicable in the default, listen and backend sections.
• timeout tarpit: this settings defines the duration for which tarpitted connections will be maintained. It is applicable in the defaults, frontend, listen, and backend sections.
• timeout tunnel: this parameter sets the maximum inactivity time on the client and server side for tunnels. This timeout applies when a bidirectional connection is established between a client and a server, and the connection remains inactive in both directions. It supersedes both the client and server timeouts once the connection becomes a tunnel. It maybe be used in the default, listen and backend sections.
• timeout http-request: this setting defines the maximum allowed time to wait for a complete HTTP request. It is applicable in the defaults, frontend, listen, and backend sections.
• timeout http-keep-alive: this sets the maximum allowed time to wait for a new HTTP request to appear. It is applicable in the defaults, frontend, listen, and backend sections.

Fix 504 Gateway Timeout Error in HAProxy

To fix the 504 gateway timeout error in HAProxy, you can adjust the value of the timeout server setting. The default value is usually 30 seconds.

Note that you have to set a timeout that meets the requirements of applications or services running in your environment. If you have long client requests such as uploading data using CSV templates with hundreds to thousands of records, consider high timeouts.

Also, if the same value applies to all other sections sections in the HAProxy configuration file, you should consider defining the settings in the default section, to avoid repetition. The following is a sample configuration with the timeout settings defined.

global
    maxconn 50000
    log	127.0.0.1 local2
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
    stats timeout 30
    user haproxy
    group haproxy
    daemon
   ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-options ssl-min-ver TLSv1.2
    ssl-dh-param-file /etc/ssl/terp/dhparam

defaults
    log	global
    mode	http
    option	httplog
    option	dontlognull
    option  forwardfor
    timeout connect 30s
    timeout client    5m
    timeout server   10m
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

listen stats
    bind *:8500
    stats enable
    stats hide-version
    stats uri /monitor
    stats realm Haproxy\ Statistics
    stats refresh 5s
    stats admin if TRUE
    stats auth fgadmin:s3cu#fd034


frontend http_in
    mode http
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/example.com.pem alpn h2,http/1.1
    redirect scheme https code 301 if !{ ssl_fc }
    http-response set-header Strict-Transport-Security max-age=63072000
    default_backend nginx_svrs

backend nginx_svrs
    mode http
    balance roundrobin
    option httpchk HEAD /
    default-server check maxconn 50000
    server nginx_svr1 10.10.1.1:80
    server nginx_svr1 10.10.1.2:80
    server nginx_svr1 10.10.1.3:80

The image below highlights the default HAProxy settings defined in the default section.

Conclusion

In this guide, I have explained the various HAProxy timeout settings. You can find a more detailed description for each setting in the HAProxy official documentation, the link is provided below. I have also shown which setting can adjusted to fix the 503 gateway timeout error in HAProxy. Have your say about this post via the feedback form below. Questions are also welcome.

References:
1. https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
2. https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration
3. https://docs.haproxy.org/