Skip to content

FPM: pm=ondemand leaves a connection in the backlog when it arrives while the last idle child is being killed #25515

Open

Description

@nealobl

Description

With pm = ondemand, a connection that arrives while the master is killing the last idle child for pm.process_idle_timeout is never handled. It stays in the listen backlog until another client connects to the same pool.

The following configuration and client:

[global]
error_log = /proc/self/fd/2

[test]
user = www-data
group = www-data
listen = /run/fpm/test.sock
listen.mode = 0666
pm = ondemand
pm.max_children = 5
pm.process_idle_timeout = 1s
pm.status_path = /status
# one client, no timeout, requesting roughly at the idle timeout interval
while true; do
  SCRIPT_NAME=/status SCRIPT_FILENAME=/status REQUEST_METHOD=GET \
    cgi-fcgi -bind -connect /run/fpm/test.sock >/dev/null
  date +%T
  sleep "$(awk -v s=$RANDOM 'BEGIN{printf "%.3f", 1+(s%2000)/1000}')"
done

Run in the official php:8.2-fpm / php:8.4-fpm images (apt-get install libfcgi-bin iproute2). Using 10 such pools with one client each makes it show up faster.

Resulted in this output:

the client stops printing; cgi-fcgi blocks forever

$ ss -xln | grep test.sock
u_str LISTEN 1      4096   /run/fpm/test.sock ...

master in epoll_wait (/proc/1/wchan: do_epoll_wait), no child process for the pool, nothing logged

With 10 pools: on 8.2.35 the first pool hung after 20 seconds; on 8.4.27, 5 of 10 pools hung within 5 minutes. With pm.process_idle_timeout = 60s and the same clients, 0 of 10 pools hung in 5 minutes (1553 requests). Opening and closing one extra connection to a stuck socket releases it immediately.

But I expected this output instead:

every request is answered; the master forks a child for the pending connection

Cause, from sapi/fpm/fpm (PHP-8.2):

  1. The idle maintenance heartbeat finds the last idle child idle for longer than pm.process_idle_timeout and sends it SIGQUIT via fpm_pctl_kill_idle_child(), setting child->idle_kill.
  2. Before that child is reaped, a new connection arrives. fpm_pctl_on_socket_accept() loops over wp->children, finds the dying child idle according to fpm_request_is_idle(), which does not look at idle_kill, and returns without forking.
  3. The child handles SIGQUIT in sig_soft_quit(), closes its listening socket and exits without accepting the connection.
  4. The ondemand event is registered with FPM_EV_EDGE, so the master gets no new event for the connection already in the backlog. It stays there until another client connects.

A possible fix is to skip children with idle_kill set in the idle check in fpm_pctl_on_socket_accept().

In production this hits monitoring agents: the Telegraf phpfpm input scrapes every 10s, the same as the default pm.process_idle_timeout, and by default waits without a timeout. On pools with no other traffic it hangs permanently. We saw this on more than 100 hosts. Busy pools recover on the next request, so visitors are rarely affected.

Possibly related: bug #69724 / GH-1308 (edge-triggered ondemand missing connections), GH-11765.

PHP Version

PHP 8.2.35, PHP 8.4.27 (reproduced); also observed in production on PHP 8.2 and 8.4

Operating System

Linux (epoll). Reproduced in the official php:*-fpm Docker images (Debian), production on Debian 13 trixie.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions