From nicblais at clkroot.net Wed Nov 28 08:01:19 2007 From: nicblais at clkroot.net (Nicolas Blais) Date: Wed Nov 28 08:20:11 2007 Subject: [Sis-SCPS-INTEREST] Connection being reset Message-ID: <20071128130119.9269F5083C@clk01.clkroot.net> Hi, I am using SCPS_RI_1_1_132 on FreeBSD 7 (both i386 and amd64 as a lib32 bin) and so far I am very impressed with the performance. Unfortunately, I am encountering problems when I pass more that 10 or so simultaneous connections on the lab. Multiple connections over a real satellite link were also an issue and I can't seem to pinpoint the actual problem. Here's the lab setup: client (test1) -> SCPS gateway -> KG-175 -> dummynet simulator <- KG-175 <- SCPS gateway -< client (test2) The simulator simulates our 2Mbit sat-link with 520-560ms delay. The lab reproduces almost exactly our actual situation when the scps gateways are not installed (IE no TCP acceleration, therefore BW is limited to TCP window size) The SCPS gateways were compiled with : --gateway=yes --gateway_larger=yes --tap=yes --low_idle=yes --low_cpu=yes --memory=512000000 and are physically very capable of handling the load. As mentionned in the handbook, I modified thread.h to have: MAX_SCPS_SOCKET 8192 and THREAD_SWAP_STACK_SIZE (131072*32) while scps_defines.h has: MAX_SCPS_SOCKET 8192 and ../include/scps.h scps_fd_set[256] Nothing special in the rfile except perhaps: AIF_BUF 65536 AIF_TP_COMPRESS 1 AIF_ACK_BEHAVE 1 BIF_TP_COMPRESS 1 And I have the following sysctl options: kern.ipc.maxsockbuf=1048576 kern.ipc.somaxconn=2048 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 When I use iperf to benchmark between test1 and test2 beyond 9 or 10 parallel TCP threads, I get broken pipes on the iperf initiating machine, and one of the gateways outputs multiple "gateway: Gateway: Connection being reset 20 0 15627" On the real network, many multiple connections are also causing similar messages. Accelerations works fine (I get 1.6mbit out of our 2mbit link) but when a big transfer is done, it seems to close ongoing connections or not allow new ones to connect and drops them. Our actual network has around 30 Windows machines on a domain (I estimate about 10 connections per machine) running shares, sql queries, local irc connections to a main server, etc... I tried to compile with --debug=yes to get more info, but I get the following error while compiling and didn't have a chance to look at it further: tp_debugPort.c: In function 'fireman': tp_debugPort.c:253: error: storage size of 'wstatus' isn't know Any help is truly appreciated, as we would really like to have this operational on our network. Thanks, Nicolas. From nicblais at clkroot.net Thu Nov 29 09:21:28 2007 From: nicblais at clkroot.net (nicblais@clkroot.net) Date: Thu Nov 29 09:50:26 2007 Subject: [Sis-SCPS-INTEREST] (no subject) Message-ID: Try #7!! (didn't see the message appear on the list, perhaps because of the attachment) As a side note to my previously reported problem, I had another problem with the SCPS gateway. When running as a bridge/tap gateway on FreeBSD 7, if you specified an MTU in your rfile and your matching real interfaces were correctly set to that MTU, the tap devices are not created with the correct MTU and therefore the bridges are incorrectly created. The console also shows : bridge0: invalid MTU for tap0 bridge1: invalid MTU for tap1 My workaround for this problem is the attached patch which sets the correct MTU as found in the rfile before creating the bridges. There are probably other ways of setting the correct MTU (such as during the tap creation in tap.c). For my aforementioned situation and without going into too much details, MTU size is important when passing through crypto devices as they usually add to the packet size and a normal MTU of 1500 will cause some crypto devices to fragment the packets when adding their own overhead. We use an MTU of 1400 on the PT side so that it will not get fragmented when going to the CT side. Hope this patch helps (and makes it to the list), Nicolas. http://www.clkroot.net/temp/patches/create_rules_tap_c.patch