Discussion:
mod_backhandunresponsive, then failure to requests
Lister
2005-02-21 01:16:51 UTC
Permalink
Configuration :
10.200.1.111 and 10.200.1.112
OS : 4.11-STABLE FreeBSD
packages (all source compiled from ports) :
apache+mod_ssl-1.3.33+2.8.22
mod_backhand-1.2.2
mod_bandwidth-2.0.5
mod_dav-1.0.3_2
mod_php4-4.3.10_2,1

Node 1 httpd.conf :
<IfModule mod_backhand.c>
UnixSocketDir /var/backhand/backhand
MulticastStats 10.200.1.111 10.200.1.255:4445,1
AcceptStats 10.200.1.0/24
</IfModule>

<Location "/backhand/">
SetHandler backhand-handler
</Location>

<Directory /usr/local/www/data/www.lauraandamber.com>
Backhand removeSelf
Backhand byBusyChildren
</Directory>

Node 2 httpd/conf :
<IfModule mod_backhand.c>
UnixSocketDir /var/backhand/backhand
MulticastStats 10.200.1.112 10.200.1.255:4445,1
AcceptStats 10.200.1.0/24
</IfModule>

<Location "/backhand/">
SetHandler backhand-handler
</Location>
Both with and with out this :
<Directory /usr/local/www/data/www.lauraandamber.com>
Backhand addSelf
</Directory>

I (re)start apache and see
[Sun Feb 20 17:10:17 2005] [notice] SIGUSR1 received. Doing graceful
restart
[Sun Feb 20 17:10:17 2005] [notice] mod_backhand -- UnixSocketDir set to
/var/backhand/backhand
[Sun Feb 20 17:10:17 2005] [notice] mod_backhand -- Broadcast
10.200.1.255:4445 added
[Sun Feb 20 17:10:17 2005] [notice] mod_backhand -- Multicast accept
10.200.1.0/24
[Sun Feb 20 17:10:17 2005] [notice] backhand_init(4859) spawning
moderator (PID 4910)
[Sun Feb 20 17:10:17 2005] [notice] mod_backhand moderator ready to go
[Sun Feb 20 17:10:18 2005] [notice] Apache/1.3.33 (Unix) PHP/4.3.10
mod_backhand/1.2.2 DAV/1.0.3 configured -- resuming normal operations
[Sun Feb 20 17:10:18 2005] [notice] Accept mutex: flock (Default: flock)

After a bunch of refreshes normal (I see the pages in each system's
access log) to the page :
http://bookpc1/www.lauraandamber.com/
netstat
tcp4 0 0 10.200.1.111.4179 10.200.1.112.http SYN_SENT
tcp4 0 0 10.200.1.111.3412 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.1875 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4199 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.2039 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3527 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4922 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4064 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3785 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4986 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3011 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.1702 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.2660 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3068 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4948 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.2797 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3074 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4851 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4449 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3046 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4777 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.http 10.200.1.90.4496 TIME_WAIT
tcp4 0 0 10.200.1.111.4098 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3982 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.1582 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.1488 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.1329 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.http 10.200.1.90.1974 TIME_WAIT
tcp4 0 0 10.200.1.111.4357 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.2882 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.2489 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3332 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.4193 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3413 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3676 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.http 10.200.1.90.1715 TIME_WAIT
tcp4 0 0 10.200.1.111.http 10.200.1.90.1416 TIME_WAIT
tcp4 0 0 10.200.1.111.4907 10.200.1.112.http TIME_WAIT
tcp4 0 0 10.200.1.111.3817 10.200.1.112.http TIME_WAIT

I see nothing in the back end's (bookpc2, .112) logs, but in the front
end's logs 9bookpc1, .111) I start to (slowly) see these :
[Sun Feb 20 16:29:55 2005] [warn] (4)Interrupted system call: connect()
timed out

Eventually, I see this :
[Sun Feb 20 14:18:09 2005] [error] [client 10.200.1.90] Request exceeded
the limit of 20 internal redirects due to probable configuration error.
Use 'LimitInternalRecursion' to increase the limit if necessary. Use
'LogLevel debug' to get a backtrace.

And / or this :
[Sun Feb 20 16:53:12 2005] [error] mod_backhand: child 4734 hung talking
to moderator. Die!
And
[Sun Feb 20 17:06:41 2005] [error] (61)Connection refused: Child 4874
failed to establish umbilical to moderator!
And
[Sun Feb 20 17:08:09 2005] [notice] child pid 4860 exit signal Broken
pipe (13)
And
message format error

This list / package appears to be have relatively low volume of
chatter. This could either mean 'really stable' or 'dead project'.
I asked on #apache, and people said it was 'in use, but not being
developed (for)'.
I would love to use this to load balance for some XML-RPC scripts
I am developing, but it has to berock solid for this. If this is not the
tool, is there something that people are using now?

TIA
Theo Schlossnagle
2005-02-21 02:21:26 UTC
Permalink
Turn of connection pooling and see if that helps your situation.

BackhandConnectionPools off
Post by Lister
I see nothing in the back end's (bookpc2, .112) logs, but in the front
connect() timed out
[Sun Feb 20 14:18:09 2005] [error] [client 10.200.1.90] Request
exceeded the limit of 20 internal redirects due to probable
configuration error. Use 'LimitInternalRecursion' to increase the
limit if necessary. Use 'LogLevel debug' to get a backtrace.
// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
// Ecelerity: fastest MTA on Earth
Lister
2005-02-21 07:10:11 UTC
Permalink
Post by Theo Schlossnagle
Turn of connection pooling and see if that helps your situation.
BackhandConnectionPools off
No change.

I also tried bumping up the kernel params :
kern.ipc.nmbclusters=16384
kern.ipc.maxsockets=16384
I think it may have pushed the envelope, but didn't fix anything.
Post by Theo Schlossnagle
Post by Lister
I see nothing in the back end's (bookpc2, .112) logs, but in the front
connect() timed out
[Sun Feb 20 14:18:09 2005] [error] [client 10.200.1.90] Request
exceeded the limit of 20 internal redirects due to probable
configuration error. Use 'LimitInternalRecursion' to increase the
limit if necessary. Use 'LogLevel debug' to get a backtrace.
// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
// Ecelerity: fastest MTA on Earth
_______________________________________________
backhand-users mailing list
http://lists.backhand.org/mailman/listinfo/backhand-users
Lister
2005-02-21 16:49:13 UTC
Permalink
Post by Theo Schlossnagle
Turn of connection pooling and see if that helps your situation.
BackhandConnectionPools off
No, wait, I had it in the <Directory> </Directory> not I put it
in the <IfModule mod_backhand.c> </IfModule> and it seems
better. Thanks.
Now it just runs really slow, on a 10 mbit lan, 3 minutes for a
109 kB.
Theo Schlossnagle
2005-02-21 14:23:17 UTC
Permalink
Post by Lister
Post by Theo Schlossnagle
Turn of connection pooling and see if that helps your situation.
BackhandConnectionPools off
No, wait, I had it in the <Directory> </Directory> not I put it
in the <IfModule mod_backhand.c> </IfModule> and it seems
better. Thanks.
Now it just runs really slow, on a 10 mbit lan, 3 minutes for a
109 kB.
Sounds like a networking problem to me. Once mod_backhand does the
proxying, it is dirt simple:

while(read from backend) { write to client }.
--
// Theo Schlossnagle
// Principal Engineer -- http://www.omniti.com/~jesus/
// Postal Engine -- http://www.postalengine.com/
// Ecelerity: fastest MTA on Earth
Lister
2005-02-21 19:03:58 UTC
Permalink
Post by Theo Schlossnagle
Post by Lister
Post by Theo Schlossnagle
Turn of connection pooling and see if that helps your situation.
BackhandConnectionPools off
No, wait, I had it in the <Directory> </Directory> not I put it
in the <IfModule mod_backhand.c> </IfModule> and it seems
better. Thanks.
Now it just runs really slow, on a 10 mbit lan, 3 minutes for a
109 kB.
Sounds like a networking problem to me. Once mod_backhand does the
while(read from backend) { write to client }.
Exactly right. Thanks.

Loading...