Fossil

clone and sync often hang or appear to hang on links with highly variable latency, sometimes doing so many retries the transfer rate is infeasibly slow and at others spinning on reads that never complete. Both git and mercurial cope well with the same network doing very similar transfers, seemingly due to their protocol design and implementatin. The fixes here improve the fossil implementation somewhat. Everything here was tested with the fossil HTTP 1.1 patch.

* Set TCP_NODELAY on the client socket

The sync protocol interleaves small control writes with bulk transfer, and with the Nagle algorithm active those small writes are held waiting for ACKs; combined with the peer's delayed ACKs this throttles throughput badly on higher-latency paths. Disabling Nagle can give a 10x throughput improvement. This is what curl and many other HTTP clients do (Git uses curl.)

* No blocking forever on TLS reads and writes

After the TLS handshake the socket is switched to non-blocking mode. ssl_send() and ssl_receive() now wait for readiness with poll() and a timeout, and abandon a connection that has stopped communicating or been reset. The blocking socket remains during the handshake so behaviour is unchanged.

* Retry on EINTR

If a signal interrupts recv()/send() we now re... (leaf check-in: e4180437 user: danshearer tags: client-timeout-fixes)

A clone or sync over HTTP/HTTPS could block when the peer stopped sending without closing the connection. The client sat in a blocking read on the socket with no timeout. The trick was to fix it without breaking the TLS handshake.

http_socket.c: set SO_RCVTIMEO and SO_SNDTIMEO (30s) on the connection socket. The fd stays in blocking mode so the TLS handshake is unaffected.

http_ssl.c: when reads time out with SO_RCVTIMEO, there is no data and BIO_should_retry() is true. Count consecutive retries without progress in ssl_send()/ssl_receive() and give up after a few. ... (check-in: 8da6e359 user: danshearer tags: client-timeout-fixes)

Fix error conditions and exceptions when Transfer-Encoding chunked, partly from drh's Claude:

* Premature EOF was accepted as a valid EOF. * strtol --> strtoll with checking, to avoid casting errors * Per RFC 7230, match "chunked" at the end of the Transfer-Encoding value rather than a substring, so the word "chunked" appearing isn't acceped as a token * make nPrior unsigned int to match blob_size() to avoid a negative offset on large replies * Report only the chunk bytes actually received, not chunk bytes from the header, so the number is correct if the connection terminates part-way. ... (leaf check-in: 441a35ce user: danshearer tags: http1-1-chunked)

Some egress filters drop all HTTP version 1.0 traffic at the Web Application Firewalls (WAF). WAFs commonly provided by CDN (Content Distribution Networks) such as CloudFlare and AWS to their customers have such controls on them, and there are understandable policy reasons for disallowing 1.0. It is almost exclusively used by old software that is often exploitable (eg ancient versions of curl). Fossil users within virtual machines with such egress filters in place get confusing messages along the lines of "Upgrade required". This is inconvenient for Fossil users, and the quick solution is to seek a git mirror for the Fossil repo if one is available, because git just works.

There is no need to upgrade the Fossil server to support 1.1 to solve the egress problem.

This is (I believe) an RFC compliant minimal implementation and it seems to work, but none of the optional extensions are supported because I don't think Fossil needs them. It does need more careful checking on the supplied chunk length because strtol is insufficient. ... (check-in: 0f8a7d60 user: danshearer tags: http1-1-chunked)