nghttp2.org

HTTP/2 C library and tools

Visualization of HTTP/2 Priority

I blogged about how HTTP/2 dependency based prioritization works in several month ago. The prioritization works fine even at that time, but there is one problem. It is a bit hard to track down and check that dependency is actually working from nghttp client logging. On the other hand, we also want a tool to measure how well server reacts to the priority hints client offers.

As a first step to tackle these problem, I added a feature to nghttp client program to output HTTP/2 transaction in well-known HAR format. We can use existing HAR file viewer to visualize the transactions and easily see how well server uses/obeys priority.

Let’s see how dependency information affects the resource transfer. Currently, nghttp prioritizes resources in 3 categories:

  1. HTML (highest)
  2. CSS (high)
  3. Javascripts (JS) (middle)
  4. images (low)

nghttp requests single HTML file and parses it while download HTML and issues additional HTTP/2 requests to server to download assets (e.g., CSS, JS and images) linked from the HTML. It does not support dynamic content yet.

I used excellent HAR file viewer in this experiment. The web content used in this experiment are taken from http2rulez, and I use nghttpx proxy server to process prioritization and content transfer scheduling.

Here is the result without priority information:

Without priority

CSS and JS are marked as shared orange and red respectively. As you can see, everything are interleaved and due to several image transfer, some of CSS and JS load times are quite slow. bootstrap.css was loaded in 417.3ms and jquery-ui.min.js was 468.9ms. Most images were loaded before them.

Let’s see how dependency based priority makes difference here. Here is the result with priority information:

With priority

The above picture clearly shows CSS and JS were loaded much faster than the version without priority. bootstrap.css, the 129.4KB largest CSS resource, was loaded in 62.6ms while jquery-ui.min.js was in 159ms. All CSS and JS are prioritized over images and this is exactly what the client asked to the server.

Nghttp2 v0.6.6

We released new nghttp2 v0.6.6. The supported HTTP/2 protocol draft version is still h2-14.

This release fixes heap-use-after free bug in priority handling.

We also fixed several bugs in priority handling code as well. The notable one is that priority is ignored when a stream that is designated as dependency parent is absent. Now it is corrected and the default priority is assigned for that case.

nghttp commnad-line client now accepts multiple -v options. Specifying -v options more than once increases verbosity.

nghttpx’s log level now accepts NOTICE level, which is now default. This feature was contributed by Lucas Pardue. The help message of -L option in nghttpx wrongly stated log level WARNING, but it is now corrected to WARN. nghttpx also implements dynamic TLS record size.

h2load now prints SSL/TLS cipher name and parameters when SSL/TLS connection is used.

Nghttp2 v0.6.5

We released new nghttp2 v0.6.5. While we merged new draft-15 features, it is compatible with draft-14 framing layer. So this release still announces h2-14 as ALPN identifier.

This release fixes heap-use-after free bug in nghttpx. The bug was found by scan-build tool.

h2load, the benchmark tool for HTTP/2 and SPDY, now has --input-file and --header options. By default, h2load takes URIs in command-line. Instead, user can store URIs in a file and give it with --input-file option to h2load. Then h2load reads URIs from that file. This is convenient if lots of URIs are needed. --header option is add additional request headers or replace pre-defined headers (e.g., :scheme). These features were contributed by Kenny (kang-yen) Peng.

nghttpx now has --strip-incoming-x-forwarded-for option. If given, it strips incoming X-Forwarded-For request header. This feature was contributed by Lucas Pardue.

nghttp, nghttpd and nghttpx now disable SSLv3 completely.

nghttpx now emits alt-svc draft-04 compliant Alt-Svc header field.

Nghttp2 v0.6.4

We released bug fix release of nghttp2 v0.6.4. It still implements h2-14 and HPACK-09.

This release fixes the application (nghttpd, nghttpx and h2load) crash with libc++ when threading is enabled. This crash was caused because of heap-use-after-free of std::mutex object. Interestingly, libstdc++ did not exhibit this issue and clang address sanitizer also did not report anything. Hopefully, this release fixes the reported crash in Mac OS X (since it is likely to use libc++), and Mac OS X users finally get multi-threaded nghttp2 applications. So try building nghttp2 without –disable-threads and run one of applications and see they do not crash.

Nghttp2 v0.6.3

We happily announce yet another release of nghttp2 v0.6.3. It still implements h2-14 and HPACK-09.

This release addresses portability/compatibility issues reported by package maintainer and users. One of the problem was caused by use of std::future. Some platform lacks std::future, which causes compile error on application under src directoy. In this release, we check that whether std::future is available or not and if not, define NOTHREADS and exclude multi threading code entirely from applications.

Surprisingly, nghttp2 did not check first SETTINGS (a part of conneciton preface) strictly. From this release, nghttp2 library code checks whether SETTINGS is received as a first frame and if other frame type is received, treats it as PROTOCOL_ERROR.

The library code itself was optimized a bit more and improved its efficiency.

Nghttp2 v0.6.2

We happily announce immediate availability of nghttp2 v0.6.2. It still implements h2-14 and HPACK-09. This release fixes memory leaks. We added nghttp2_option_set_recv_client_preface() function. Previously, nghttp2 library only handles HTTP/2 frames and does not recognize first 24 bytes of client connection preface. This design choice is done due to the fact that server may want to detect the application protocol based on first few bytes on clear text communication. But for simple servers which only speak HTTP/2, it is easier for developers if nghttp2 library takes care of client connection preface as well. If this option is enabled, nghttp2 library checks first 24 bytes client connection preface. If it is not a valid one, nghttp2_session_recv() and nghttp2_session_mem_recv() will return new error code NGHTTP2_ERR_BAD_PREFACE, which is fatal error. By default, the option is turned off for backward compatibility.

The other thing worth mentioning is that now nghttp2_stream_resume_deferred_data() does not fail if data is deferred by flow control.

Previously processing incoming connection level WINDOW_UPDATE frame is quite expensive, but it is fixed and now 2x faster than v0.6.1.

We introduced the experimental libnghttp2_asio C++ library. It is built on top of libnghttp2 and intended to provide higher level APIs to build HTTP/2 client/server easily. Currently, only server APIs are provided. It depends on Boost::Asio and Boost::Thread libraries. For example, the rather useless simple HTTP/2 server which returns “hello, world” for any HTTP request is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#include <nghttp2/asio_http2.h>

using namespace nghttp2::asio_http2;
using namespace nghttp2::asio_http2::server;

int main(int argc, char **argv)
{
  http2 server;

  server.listen
    ("127.0.0.1", 3000,
     [](std::shared_ptr<request> req, std::shared_ptr<response> res)
     {
       res->write_head(200);
       res->end("hello, world");
     });
}

For more details, consult library documentation.

We added examples/tiny-nghttpd server, which is stripped faster version of nghttpd. Its purpose is measure performance bottleneck in libnghttp2 code, avoding any overhead in external I/O library. Currently it requires epoll, so it can be built only on linux.

Host Lucid Erlang HTTP/2 Server

nghttp2.org now hosts Lucid HTTP/2 server written in Erlang. The access endpoint is https://nghttp2.org:3456/. Currently it returns simple html content including request headers. It only speaks h2-14.

Since Erlang SSL module does not support AEAD cipher suites, Firefox nightly refuses to connect to this server. To workaround this, open about:config in Firefox and set network.http.spdy.enforce-tls-profile to false.

Nghttp2 v0.6.1

We just released nghttp2 v0.6.1. This release fixes compile error in bundled application code on OSX. No library code has been changed since v0.6.0.

Nghttp2 v0.6.0

Finally, we released nghttp2 v0.6.0! It implements h2-14 and header compression HPACK 09.

In this release, we decided to hide the details of nghttp2_session_callbacks struct. The reason of this decision is to avoid so name bump each time we add new callback. Actually, we added 2 new callbacks in this release. We expect more to come, so it is a good time to make API open for extension.

We successfully performed interop testing with Firefox, Jetty and Twitter.

If you are shrpx users from spdylay project, this is a good time to migrate to nghttpx, which supports SPDY proxy as well.

Anyway, the every release for OSS project is happy time. Have fun!

Update to H2-14

Today We updated nghttp2 to support latest HTTP/2 draft, h2-14. The source code is available on github. This server is advertize h2-14 now.

Here is a quick summary of the changes since h2-13:

  • Frame length field was expanded to 24 bits.
  • The pseudo headers (aka, colon (:) headers) handling is tightened up. Now unexpected pseudo headers are subject to stream error.
  • WINDOW_UPDATE with 0 window-size is now treated as error.
  • END_SEGMENT flag was removed.
  • SETTINGS_MAX_FRAME_SIZE and SETTINGS_MAX_HEADER_LIST_SIZE were added.
  • 1xx status codes, except for 101, are now supported.
  • Removed 0x00 concatenation rule.
  • HPACK: reference set was removed.
  • HPACK: static header table is now in front of dynamic header table.
  • HPACK: No copy when static header table entry is referenced.