Repository navigation
Description
Description
With pm = ondemand, a connection that arrives while the master is killing the last idle child for pm.process_idle_timeout is never handled. It stays in the listen backlog until another client connects to the same pool.
The following configuration and client:
[global]
error_log = /proc/self/fd/2
[test]
user = www-data
group = www-data
listen = /run/fpm/test.sock
listen.mode = 0666
pm = ondemand
pm.max_children = 5
pm.process_idle_timeout = 1s
pm.status_path = /status# one client, no timeout, requesting roughly at the idle timeout interval
while true; do
SCRIPT_NAME=/status SCRIPT_FILENAME=/status REQUEST_METHOD=GET \
cgi-fcgi -bind -connect /run/fpm/test.sock >/dev/null
date +%T
sleep "$(awk -v s=$RANDOM 'BEGIN{printf "%.3f", 1+(s%2000)/1000}')"
doneRun in the official php:8.2-fpm / php:8.4-fpm images (apt-get install libfcgi-bin iproute2). Using 10 such pools with one client each makes it show up faster.
Resulted in this output:
the client stops printing; cgi-fcgi blocks forever
$ ss -xln | grep test.sock
u_str LISTEN 1 4096 /run/fpm/test.sock ...
master in epoll_wait (/proc/1/wchan: do_epoll_wait), no child process for the pool, nothing logged
With 10 pools: on 8.2.35 the first pool hung after 20 seconds; on 8.4.27, 5 of 10 pools hung within 5 minutes. With pm.process_idle_timeout = 60s and the same clients, 0 of 10 pools hung in 5 minutes (1553 requests). Opening and closing one extra connection to a stuck socket releases it immediately.
But I expected this output instead:
every request is answered; the master forks a child for the pending connection
Cause, from sapi/fpm/fpm (PHP-8.2):
- The idle maintenance heartbeat finds the last idle child idle for longer than
pm.process_idle_timeoutand sends it SIGQUIT viafpm_pctl_kill_idle_child(), settingchild->idle_kill. - Before that child is reaped, a new connection arrives.
fpm_pctl_on_socket_accept()loops overwp->children, finds the dying child idle according tofpm_request_is_idle(), which does not look atidle_kill, and returns without forking. - The child handles SIGQUIT in
sig_soft_quit(), closes its listening socket and exits without accepting the connection. - The ondemand event is registered with
FPM_EV_EDGE, so the master gets no new event for the connection already in the backlog. It stays there until another client connects.
A possible fix is to skip children with idle_kill set in the idle check in fpm_pctl_on_socket_accept().
In production this hits monitoring agents: the Telegraf phpfpm input scrapes every 10s, the same as the default pm.process_idle_timeout, and by default waits without a timeout. On pools with no other traffic it hangs permanently. We saw this on more than 100 hosts. Busy pools recover on the next request, so visitors are rarely affected.
Possibly related: bug #69724 / GH-1308 (edge-triggered ondemand missing connections), GH-11765.
PHP Version
PHP 8.2.35, PHP 8.4.27 (reproduced); also observed in production on PHP 8.2 and 8.4
Operating System
Linux (epoll). Reproduced in the official php:*-fpm Docker images (Debian), production on Debian 13 trixie.
Activity