What's wrong with this code, part 5: The answers

re: What's wrong with this code, part 5: The answers

Keith Moore [exmsft] — Thu, 05 Aug 2004 21:27:00 GMT

FWIW: A couple of bits of socket trivia.

1. The loop in TransmitTcpBuffer() is unnecessary if called with a stream-oriented socket (like, say, TCP). Calling WriteFile(), send(), or WSASend() on a stream-oriented socket will either a) completely send the entire buffer, or b) fail.

2. Zero-byte sends may or may not actually cause packets to hit the wire, depending on the underlying protocol. For TCP, it's a noop. For UDP, it sends a zero-byte datagram. For message-oriented protocols (like SPX), it sends a zero-byte message.

re: What's wrong with this code, part 5: The answers

Larry Osterman — Thu, 05 Aug 2004 21:30:00 GMT

Thanks Keith, trust the guy who wrote the code to know the REAL answer :)

Btw, the WriteFile()/send() behavior does NOT apply in general, even with Henry's TCP.

For Exchange, we had issues with partial writes on Win16 and (I believe) Win9x, the RPC runtime didn't handle partial writes properly and that caused some "interesting" bugs.

re: What's wrong with this code, part 5: The answers

Anonymous — Thu, 05 Aug 2004 22:39:00 GMT

Thanks for your detailed answer. I have a question.

Doesn't Windows have a per socket send/receive buffer? Doesn't send() return after the data is stored in the buffer (instead of waiting for data to leave the wire).

What about receive, when a TCP packet is received from the adapter doesn't it go to the receive buffer first? How big is the buffer? How can the size be changed?

re: What's wrong with this code, part 5: The answers

Larry Osterman — Thu, 05 Aug 2004 22:46:00 GMT

Yes it has a per-socket send/receive buffer.

But most networking APIs (and in this context writefile is a networking API) guarantee that the receiver has acknowledged receipt of the data before the write completes. Otherwise you could lose data and not have any way of knowing it.

There ARE varients that don't guarantee that data is received, but that's clearly spelled out in the API.

re: What's wrong with this code, part 5: The answers

Norman Diamond — Fri, 06 Aug 2004 00:48:00 GMT

8/5/2004 3:39 PM Anonymous

> I have a question.

Give or take an order of magnitude.

> Doesn't send() return after the data is
> stored in the buffer

Surely that part of it depends on the protocol and the socket options.

8/5/2004 3:46 PM Larry Osterman

> But most networking APIs [...] guarantee

OK, UDP isn't "most", but are the proportions really so unbalanced? This makes me wonder if people who argued for putting acknowledgement in the original Ethernet design might have been right.

re: What's wrong with this code, part 5: The answers

Larry Osterman — Fri, 06 Aug 2004 00:52:00 GMT

Norman, you're right. I believe that you can configure sockets to return before the data is delivered, but that's NOT the default behavior.

And yes, I think that the proportions are that unbalanced. Networking is fundimentally an unreliable medium, but programmers want reliable transfer mechanisms. So the networking guys designed their APIs to be as reliable as possible. If you want unreliable, you need to ask for it.

re: What's wrong with this code, part 5: The answers

Larry Osterman — Fri, 06 Aug 2004 00:56:00 GMT

Norman, one example of a "you can shoot yourself in the foot if you want" is the Netbios sendnoack option - that would allow sends to complete before the receiver had completed them, but you had to handle the APIs specially (I don't remember the details now, it was about 15 years ago when I dealt with them).

re: What's wrong with this code, part 5: The answers

Random Reader — Fri, 06 Aug 2004 05:26:00 GMT

> I believe that you can configure sockets to return before the data is delivered, but that's NOT the default behavior.

This doesn't make any sense. There would be no point whatsoever to the socket's send buffer if this were the case. A blocking send() would simply lock the application's buffer until done. The docs imply that they don't work this way.

Even send() explicitly states "The successful completion of a send does not indicate that the data was successfully delivered." (http://msdn.microsoft.com/library/en-us/winsock/winsock/send_2.asp)

WriteFile() shouldn't be special in this regard either, since you can't even guarantee it writes to a _file_ unless you disable all buffering on the handle, or flush it afterward. Why would the network be special?

re: What's wrong with this code, part 5: The answers

Random Reader — Fri, 06 Aug 2004 06:22:00 GMT

I followed this up with an extremely ugly and utterly non-robust test program:

#include <stdio.h>
#include <winsock2.h>
#include <windows.h>

int main(void)
{
SOCKET s;
WSADATA wsa;
struct sockaddr_in sa = {0};

sa.sin_family = AF_INET;
sa.sin_port = htons(9000);
sa.sin_addr.S_un.S_addr = inet_addr("192.168.0.6");

WSAStartup(MAKEWORD(2,2), &wsa);
s = socket(AF_INET, SOCK_STREAM, 0);
connect(s, (struct sockaddr *)&sa, sizeof(sa));
printf("connect() finished\n");
getchar();
send(s, "123", 3, 0);
printf("send() finished\n");
getchar();
closesocket(s);
WSACleanup();

return 0;
}

I used Sysinternals' TCPView to monitor things, since I obviously left error checking out of the code. After the connection was made, I unplugged the target machine's network cable and tapped enter. The send() completed immediately.

What was fun to see was when I tapped enter again to close the socket and exit the program, and only then plugged the target back in. Shortly afterward, it received the data :)

What puzzles me at the moment is that I couldn't make WriteFile() work as a drop-in replacement for send() there. It always reported ERROR_INVALID_PARAMETER, so I must be missing something.

On a different note, why does a connected socket appear to hold the same local port open for listening as well? For example, in one connection, local 192.168.0.5:6747 was ESTABLISHED to 192.168.0.6:9000, and local 0.0.0.0:6747 was LISTENING. It's not possible to connect to, but I'm curious why it shows up at all.

re: What's wrong with this code, part 5: The answers

StefanG — Fri, 06 Aug 2004 09:10:00 GMT

Sorry Larry, Your blog is usually quite excellent, but in this case there is definitely
something wrong with what you are saying.

Assuming that no special socket options have been set, there are no serious perf problems with the code.

Each socket by default has a kernel buffer of 8192 bytes. If the data you are writing in the WriteFile fits in the buffer
WriteFile returns immediately.

The behaviour you describe will only happen if you have disabled the socket buffer by setting SO_SNDBUF to 0

An article that describes these issues very well can be found at
http://support.microsoft.com/default.aspx?scid=kb;en-us;214397

I have made some quick tests to verify what I am saying. I used pcattcp.exe which can be downloaded from
http://www.pcausa.com/Utilities/ttcpdown1.htm

Sending data in 4096-byte chunks with default options: 0.36 ms/call
SO_SNDBUF=0: 7.29 ms/call
SO_SNDBUF=0, TCP_NODELAY=1: 7.21 ms/call

H:\ttcp>pcattcp -t -l 4096 srmutv
PCAUSA Test TCP Utility V2.01.01.07
TCP Transmit Test
Remote Host : srmutv
Transmit : TCP -> 192.71.83.122:5001
Buffer Size : 4096; Alignment: 16384/0
TCP_NODELAY : DISABLED (0)
Connect : Connected to 192.71.83.122:5001
Send Mode : Send Pattern; Number of Buffers: 2048
Statistics : TCP -> 192.71.83.122:5001
8388608 bytes in 0.72 real seconds = 11362.00 KB/sec +++
numCalls: 2048; msec/call: 0.36; calls/sec: 2840.50

H:\ttcp>pcattcp -t -l 4096 -b 0 srmutv
PCAUSA Test TCP Utility V2.01.01.07
TCP Transmit Test
Remote Host : srmutv
Transmit : TCP -> 192.71.83.122:5001
Buffer Size : 4096; Alignment: 16384/0
SO_SNDBUF : 0
TCP_NODELAY : DISABLED (0)
Connect : Connected to 192.71.83.122:5001
Send Mode : Send Pattern; Number of Buffers: 2048
Statistics : TCP -> 192.71.83.122:5001
8388608 bytes in 14.57 real seconds = 562.21 KB/sec +++
numCalls: 2048; msec/call: 7.29; calls/sec: 140.55

H:\ttcp>pcattcp -t -l 4096 -b 0 -D srmutv
PCAUSA Test TCP Utility V2.01.01.07
TCP Transmit Test
Remote Host : srmutv
Transmit : TCP -> 192.71.83.122:5001
Buffer Size : 4096; Alignment: 16384/0
SO_SNDBUF : 0
TCP_NODELAY : ENABLED (1)
Connect : Connected to 192.71.83.122:5001
Send Mode : Send Pattern; Number of Buffers: 2048
Statistics : TCP -> 192.71.83.122:5001
8388608 bytes in 14.42 real seconds = 568.06 KB/sec +++
numCalls: 2048; msec/call: 7.21; calls/sec: 142.02

re: What's wrong with this code, part 5: The answers

Question — Fri, 06 Aug 2004 14:31:00 GMT

Wait a sec, is the Nagling algorithm turned on by default? If not, how do I turn it on for a socket? (I have an app that sends multiple small packets)

re: What's wrong with this code, part 5: The answers

Larry Osterman — Fri, 06 Aug 2004 15:01:00 GMT

I'm calling in the big guns for help (asked the winsock lead for comments). He may comment later. But here's my comments off the top of my head.

Question: As far as I know, nagling is ON by default. The easy way to tell is to run Stephan's test with 10 byte writes and look at the results from a remote machine - if the data is transmitted on the wire immediately in 10 byte writes nagling's disabled. If it's transmitted in larger writes it's enabled.

Stephan, did you try the same test with an odd number of frames with a buffer GREATER than 8192 bytes that takes an odd number of frames? Also, to see this behavior you need TWO machines - one transmits the data, the other measures the throughput. Having netmon (or etherial) capture the data would also show the behavior.

It's possible that http.sys sets SO_SNDBUF to 0, which is why we didn't see this in WMC earlier this week.

But the behavior is VERY real, and it's burned more people than I can think of.

re: What's wrong with this code, part 5: The answers

Anonymous — Fri, 06 Aug 2004 16:48:00 GMT

Assuming Larry is right, and assuming default send buffers, isn't it difficult to make sure the sent data is acknowledged. Say one thread sends 100 byes using WriteFile() and waits for ack, and another thread sends 200 bytes on the same socket. Now on a stream protocol like TCP, the driver has to mark the boundary after the first send and monitor it, and it is painful to implement that feature. This gets complicated with overlapped I/O.

Coming from the unix world, I think the only way to guarantee data delivery is to set the send buf to 0 bytes, even that probably only waits for data to leave the wire, doesn't wait for the ack (I am not sure though).

This an interesting discussion, Thanks StefanG and Random Reader for your thougts.

re: What's wrong with this code, part 5: The answers

Larry Osterman — Fri, 06 Aug 2004 17:00:00 GMT

From the winsock wizards, when I asked them:
Yes, Winsock has implemented this default 8k buffer exactly for this reason - make such naive apps work. After all most of the apps out there send relatively small packets and cost of extra copy is cheaper that locking the buffer and sending directly from application space. And for those that need to send files, there is a TransmitFile API that implements proper dual buffer technique.

So my app would be broken given a larger buffer size. I'm still trying to get confirmation that http.sys sets SO_SNDBUF to 0, which would complete my understanding of why I messed up so badly.

As I said: This IS a real problem, and it shows up with great regularity, which is why I wanted to write about it. It was my mistake in not actually verifying this with a real application :(

The consequences of ignoring nagling and delayed acks

Larry Osterman's WebLog — Fri, 06 Aug 2004 21:33:00 GMT

re: What's wrong with this code, part 5: The answers

Simon Cooke [exMSFT] — Fri, 06 Aug 2004 19:05:00 GMT

Anon: I set the buffer to 0 on the high-throughput networking stuff I'm doing. Couple it with overlapped io and it allows one to at least get a certain amount of control over network throttling.

re: What's wrong with this code, part 5: The answers

Random Reader — Sat, 07 Aug 2004 09:15:00 GMT

Ahh, that explains it.

And to follow up on my WriteFile() trouble, it turns out that socket() always creates an overlapped socket, and WriteFile() requires a valid OVERLAPPED structure to work on such handles. The fix was to simply use WSASocket() with no flags.

Despite any mistakes, it's good to get all this stuff cleared up.

re: What's wrong with this code, part 5: The answers

pUnk — Mon, 09 Aug 2004 03:26:00 GMT

1568? Regular old Ethernet is 1518 from Ethernet header to/including FCS.

re: What's wrong with this code, part 5: The answers

Larry Osterman — Mon, 09 Aug 2004 16:19:00 GMT

pUnk, I could have sworn that 1568 was max ethernet frame size, but clearly that's not right.