Fix improper cache key on write for IPRO of resources with PageSpeed params

We need to be writing with the base name, with params stripped. This was causing a test to "flake" under valgrind, since with it we got to the test showing the bug slowly enough for the entry written out by an earlier test under proper URL to expire.
trunk-tracking: update from r3872 to r3895
2014-04-02 18:41:08 +02:00 · 2014-04-02 18:39:08 +02:00 · 2014-03-31 10:35:05 -04:00 · 2014-03-28 22:37:31 +01:00 · 2014-03-27 10:59:13 -04:00 · 2014-03-26 08:53:50 +01:00
19 changed files with 1870 additions and 672 deletions
@@ -37,21 +37,22 @@ recompiling Tengine](https://github.com/pagespeed/ngx_pagespeed/wiki/Using-ngx_p

   ```bash
   $ cd ~
-   $ wget https://github.com/pagespeed/ngx_pagespeed/archive/release-1.7.30.1-beta.zip
-   $ unzip release-1.7.30.1-beta.zip # or unzip release-1.7.30.1-beta
-   $ cd ngx_pagespeed-release-1.7.30.1-beta/
-   $ wget https://dl.google.com/dl/page-speed/psol/1.7.30.1.tar.gz
-   $ tar -xzvf 1.7.30.1.tar.gz # expands to psol/
+   $ wget https://github.com/pagespeed/ngx_pagespeed/archive/v1.7.30.4-beta.zip
+   $ unzip v1.7.30.4-beta.zip # or unzip v1.7.30.4-beta
+   $ cd ngx_pagespeed-1.7.30.4-beta/
+   $ wget https://dl.google.com/dl/page-speed/psol/1.7.30.4.tar.gz
+   $ tar -xzvf 1.7.30.4.tar.gz # expands to psol/
   ```

 3. Download and build nginx:

   ```bash
+   $ cd ~
   $ # check http://nginx.org/en/download.html for the latest version
-   $ wget http://nginx.org/download/nginx-1.4.3.tar.gz
-   $ tar -xvzf nginx-1.4.3.tar.gz
-   $ cd nginx-1.4.3/
-   $ ./configure --add-module=$HOME/ngx_pagespeed-release-1.7.30.1-beta
+   $ wget http://nginx.org/download/nginx-1.4.6.tar.gz
+   $ tar -xvzf nginx-1.4.6.tar.gz
+   $ cd nginx-1.4.6/
+   $ ./configure --add-module=$HOME/ngx_pagespeed-1.7.30.4-beta
   $ make
   $ sudo make install
   ```
@@ -94,7 +95,7 @@ To confirm that the module is loaded, fetch a page and check that you see the

 ```bash
 $ curl -I 'http://localhost:8050/some_page/' | grep X-Page-Speed
-X-Page-Speed: 1.7.30.1-...
+X-Page-Speed: 1.7.30.4-...
 ```

 Looking at the source of a few pages you should see various changes, such as
@@ -27,8 +27,8 @@ if [ "$mod_pagespeed_dir" = "unset" ] ; then
    echo "   You need to separately download the pagespeed library:"
    echo ""
    echo "     $ cd /path/to/ngx_pagespeed"
-    echo "     $ wget https://dl.google.com/dl/page-speed/psol/1.7.30.1.tar.gz"
-    echo "     $ tar -xzvf 1.7.30.1.tar.gz # expands to psol/"
+    echo "     $ wget https://dl.google.com/dl/page-speed/psol/1.7.30.4.tar.gz"
+    echo "     $ tar -xzvf 1.7.30.4.tar.gz # expands to psol/"
    echo ""
    echo "   Or see the installation instructions:"
    echo "     https://github.com/pagespeed/ngx_pagespeed#how-to-build"
@@ -89,9 +89,7 @@ if [ "$uname_arch" = "i686" ]; then
  FLAG_MARCH='-march=i686'
 fi

-# Building with HTTPS fetching enabled pulls in a version of OpenSSL that causes
-# linker errors, so disable it here.
-CFLAGS="$CFLAGS -DSERF_HTTPS_FETCHING=0 $FLAG_MARCH"
+CFLAGS="$CFLAGS $FLAG_MARCH"

 case "$NGX_GCC_VER" in
  4.8*)
@@ -128,17 +126,10 @@ ngx_feature_path="$pagespeed_include"

 if $build_from_source ; then
  psol_library_binaries="\
-    $mod_pagespeed_dir/net/instaweb/automatic/pagespeed_automatic.a \
-    $mod_pagespeed_dir/out/$buildtype/obj.target/third_party/serf/libserf.a \
-    $mod_pagespeed_dir/out/$buildtype/obj.target/third_party/aprutil/libaprutil.a \
-    $mod_pagespeed_dir/out/$buildtype/obj.target/third_party/apr/libapr.a"
+    $mod_pagespeed_dir/net/instaweb/automatic/pagespeed_automatic.a"
 else
  psol_library_dir="$ngx_addon_dir/psol/lib/$buildtype/$os_name/$arch_name"
-  psol_library_binaries="\
-    $psol_library_dir/pagespeed_automatic.a \
-    $psol_library_dir/libserf.a \
-    $psol_library_dir/libaprutil.a \
-    $psol_library_dir/libapr.a"
+  psol_library_binaries="$psol_library_dir/pagespeed_automatic.a"
 fi

 pagespeed_libs="-lstdc++ $psol_library_binaries -lrt -pthread -lm"
@@ -190,13 +181,7 @@ if [ $ngx_found = yes ]; then
    $ps_src/ngx_rewrite_driver_factory.cc \
    $ps_src/ngx_rewrite_options.cc \
    $ps_src/ngx_server_context.cc \
-    $ps_src/ngx_url_async_fetcher.cc \
-    $mod_pagespeed_dir/out/$buildtype/obj/gen/data2c_out/instaweb/net/instaweb/apache/install/mod_pagespeed_example/mod_pagespeed_console_out.cc \
-    $mod_pagespeed_dir/out/$buildtype/obj/gen/data2c_out/instaweb/net/instaweb/apache/install/mod_pagespeed_example/mod_pagespeed_console_css_out.cc \
-    $mod_pagespeed_dir/out/$buildtype/obj/gen/data2c_out/instaweb/net/instaweb/apache/install/mod_pagespeed_example/mod_pagespeed_console_html_out.cc \
-    $mod_pagespeed_dir/net/instaweb/system/add_headers_fetcher.cc \
-    $mod_pagespeed_dir/net/instaweb/system/loopback_route_fetcher.cc \
-    $mod_pagespeed_dir/net/instaweb/system/serf_url_async_fetcher.cc"
+    $ps_src/ngx_url_async_fetcher.cc"

  # Make pagespeed run immediately before gzip.
  HTTP_FILTER_MODULES=$(echo $HTTP_FILTER_MODULES |\
@@ -206,9 +191,11 @@ if [ $ngx_found = yes ]; then
    sed "s/$HTTP_GZIP_FILTER_MODULE/ngx_pagespeed_etag_filter $HTTP_GZIP_FILTER_MODULE/")
  CORE_LIBS="$CORE_LIBS $pagespeed_libs"
  CORE_INCS="$CORE_INCS $pagespeed_include"
+  echo "List of modules (in reverse order of applicability): "$HTTP_FILTER_MODULES
 else
  cat << END
-$0: error: module ngx_pagespeed requires the pagespeed optimization library
+$0: error: module ngx_pagespeed requires the pagespeed optimization library.
+Look in obj/autoconf.err for more details.
 END
  exit 1
 fi
@@ -106,11 +106,10 @@ ngx_int_t NgxBaseFetch::CollectAccumulatedWrites(ngx_chain_t** link_ptr) {
 ngx_int_t NgxBaseFetch::CollectHeaders(ngx_http_headers_out_t* headers_out) {
  const ResponseHeaders* pagespeed_headers = response_headers();

-  // TODO(chaizhenhua): Add and check.
-  // if (content_length_known()) {
-  //   headers_out->content_length = NULL;
-  //   headers_out->content_length_n = content_length();
-  // }
+  if (content_length_known()) {
+     headers_out->content_length = NULL;
+     headers_out->content_length_n = content_length();
+  }

  return copy_response_headers_to_ngx(request_, *pagespeed_headers,
                                      preserve_caching_headers_);
@@ -24,6 +24,10 @@
 //  - The read handler parses the response. Add the response to the buffer at
 //    last.

+extern "C" {
+#include <nginx.h>
+}
+
 #include "ngx_fetch.h"
 #include "net/instaweb/util/public/basictypes.h"
 #include "base/logging.h"
@@ -66,7 +70,8 @@ namespace net_instaweb {
        fetch_start_ms_(0),
        fetch_end_ms_(0),
        done_(false),
-        content_length_(0) {
+        content_length_(-1),
+        content_length_known_(false) {
            ngx_memzero(&url_, sizeof(url_));
            log_ = log;
            pool_ = NULL;
@@ -138,19 +143,26 @@ namespace net_instaweb {
    // The host is either a domain name or an IP address.  First check
    // if it's a valid IP address and only if that fails fall back to
    // using the DNS resolver.
-    GoogleString s_ipaddress(reinterpret_cast<char*>(url_.host.data),
-                             url_.host.len);
+
+    // Maybe we have a Proxy.
+    ngx_url_t* tmp_url = &url_;
+    if (0 != fetcher_->proxy_.url.len) {
+      tmp_url = &fetcher_->proxy_;
+    }
+
+    GoogleString s_ipaddress(reinterpret_cast<char*>(tmp_url->host.data),
+                             tmp_url->host.len);
    ngx_memzero(&sin_, sizeof(sin_));
    sin_.sin_family = AF_INET;
-    sin_.sin_port = htons(url_.port);
+    sin_.sin_port = htons(tmp_url->port);
    sin_.sin_addr.s_addr = inet_addr(s_ipaddress.c_str());

    if (sin_.sin_addr.s_addr == INADDR_NONE) {
      // inet_addr returned INADDR_NONE, which means the hostname
      // isn't a valid IP address.  Check DNS.
      ngx_resolver_ctx_t temp;
-      temp.name.data = url_.host.data;
-      temp.name.len = url_.host.len;
+      temp.name.data = tmp_url->host.data;
+      temp.name.len = tmp_url->host.len;
      resolver_ctx_ = ngx_resolve_start(fetcher_->resolver_, &temp);
      if (resolver_ctx_ == NULL || resolver_ctx_ == NGX_NO_RESOLVER) {
        // TODO(oschaaf): this spams the log, but is useful in the fetcher's
@@ -162,9 +174,13 @@ namespace net_instaweb {
      }

      resolver_ctx_->data = this;
-      resolver_ctx_->name.data = url_.host.data;
-      resolver_ctx_->name.len = url_.host.len;
+      resolver_ctx_->name.data = tmp_url->host.data;
+      resolver_ctx_->name.len = tmp_url->host.len;
+
+#if (nginx_version < 1005008)
      resolver_ctx_->type = NGX_RESOLVE_A;
+#endif
+
      resolver_ctx_->handler = NgxFetchResolveDone;
      resolver_ctx_->timeout = fetcher_->resolver_timeout_;

@@ -255,37 +271,14 @@ namespace net_instaweb {
      return false;
    }
    str_url_.copy(reinterpret_cast<char*>(url_.url.data), str_url_.length(), 0);
-    size_t scheme_offset;
-    u_short port;
-    if (ngx_strncasecmp(url_.url.data, reinterpret_cast<u_char*>(
-                                     const_cast<char*>("http://")), 7) == 0) {
-      scheme_offset = 7;
-      port = 80;
-    } else if (ngx_strncasecmp(url_.url.data, reinterpret_cast<u_char*>(
-                                    const_cast<char*>("https://")), 8) == 0) {
-      scheme_offset = 8;
-      port = 443;
-    } else {
-      scheme_offset = 0;
-      port = 80;
-    }

-    url_.url.data += scheme_offset;
-    url_.url.len -= scheme_offset;
-    url_.default_port = port;
-    // See: http://lxr.evanmiller.org/http/source/core/ngx_inet.c#L875
-    url_.no_resolve = 0;
-    url_.uri_part = 1;
-
-    if (ngx_parse_url(pool_, &url_) == NGX_OK) {
-      return true;
-    }
-    return false;
+    return NgxUrlAsyncFetcher::ParseUrl(&url_, pool_);
  }

  // Issue a request after the resolver is done
  void NgxFetch::NgxFetchResolveDone(ngx_resolver_ctx_t* resolver_ctx) {
    NgxFetch* fetch = static_cast<NgxFetch*>(resolver_ctx->data);
+    NgxUrlAsyncFetcher* fetcher = fetch->fetcher_;
    if (resolver_ctx->state != NGX_OK) {
      if (fetch->timeout_event() != NULL && fetch->timeout_event()->timer_set) {
        ngx_del_timer(fetch->timeout_event());
@@ -299,9 +292,25 @@ namespace net_instaweb {
      return;
    }
    ngx_memzero(&fetch->sin_, sizeof(fetch->sin_));
+
+#if (nginx_version < 1005008)
+    fetch->sin_.sin_addr.s_addr = resolver_ctx->addrs[0];
+#else
+
+    struct sockaddr_in *sin;
+    sin = reinterpret_cast<struct sockaddr_in *>(
+        resolver_ctx->addrs[0].sockaddr);
+    fetch->sin_.sin_family = sin->sin_family;
+    fetch->sin_.sin_addr.s_addr = sin->sin_addr.s_addr;
+#endif
+
    fetch->sin_.sin_family = AF_INET;
    fetch->sin_.sin_port = htons(fetch->url_.port);
-    fetch->sin_.sin_addr.s_addr = resolver_ctx->addrs[0];
+
+    // Maybe we have Proxy
+    if (0 != fetcher->proxy_.url.len) {
+      fetch->sin_.sin_port = htons(fetcher->proxy_.port);
+    }

    char* ip_address = inet_ntoa(fetch->sin_.sin_addr);

@@ -333,7 +342,13 @@ namespace net_instaweb {
    bool have_host = false;
    GoogleString port;

-    size = sizeof("GET ") - 1 + url_.uri.len + sizeof(" HTTP/1.0\r\n") - 1;
+    const char* method = request_headers->method_string();
+    size_t method_len = strlen(method);
+
+    size = (method_len +
+            1 /* for the space */ +
+            url_.uri.len +
+            sizeof(" HTTP/1.0\r\n") - 1);

    for (int i = 0; i < request_headers->NumAttributes(); i++) {
      // if no explicit host header is given in the request headers,
@@ -361,7 +376,8 @@ namespace net_instaweb {
      return NGX_ERROR;
    }

-    out_->last = ngx_cpymem(out_->last, "GET ", 4);
+    out_->last = ngx_cpymem(out_->last, method, method_len);
+    out_->last = ngx_cpymem(out_->last, " ", 1);
    out_->last = ngx_cpymem(out_->last, url_.uri.data, url_.uri.len);
    out_->last = ngx_cpymem(out_->last, " HTTP/1.0\r\n", 11);

@@ -469,9 +485,16 @@ namespace net_instaweb {
      }

      if (n == 0) {
-        // connection is closed prematurely by remote server,
-        // or the content-length was 0
-        fetch->CallbackDone(fetch->content_length_ == 0);
+        // If the content length was not known, we assume that we have read
+        // all if we at least parsed the headers.
+        // If we do know the content length, having a mismatch on the bytes read
+        // will be interpreted as an error.
+        if (fetch->content_length_known_) {
+          fetch->CallbackDone(fetch->content_length_ == fetch->bytes_received_);
+        } else {
+          fetch->CallbackDone(fetch->parser_.headers_complete());
+        }
+
        return;
      } else if (n > 0) {
        fetch->in_->pos = fetch->in_->start;
@@ -538,13 +561,21 @@ namespace net_instaweb {
    if (n > size) {
      return false;
    } else if (fetch->parser_.headers_complete()) {
-      int64 content_length = -1;
-      fetch->async_fetch_->response_headers()->FindContentLength(
-          &content_length);
-      fetch->content_length_ = content_length;
-      if (fetch->fetcher_->track_original_content_length()) {
+      if (fetch->async_fetch_->response_headers()->FindContentLength(
+              &fetch->content_length_)) {
+        if (fetch->content_length_ < 0) {
+          fetch->message_handler_->Message(
+              kError, "Negative content-length in response header");
+          return false;
+        } else {
+          fetch->content_length_known_ = true;
+        }
+      }
+
+      if (fetch->fetcher_->track_original_content_length()
+         && fetch->content_length_known_) {
        fetch->async_fetch_->response_headers()->SetOriginalContentLength(
-            content_length);
+            fetch->content_length_);
      }

      fetch->in_->pos += n;
@@ -563,11 +594,12 @@ namespace net_instaweb {
      return true;
    }

-    fetch->bytes_received_add(static_cast<int64>(size));
+    fetch->bytes_received_add(size);
+
    if (fetch->async_fetch_->Write(StringPiece(data, size),
        fetch->message_handler())) {
-      fetch->content_length_ -= size;
-      if (fetch->content_length_ <= 0) {
+      if (fetch->content_length_known_ &&
+          fetch->bytes_received_ == fetch->content_length_) {
        fetch->done_ = true;
      }
      return true;
@@ -121,12 +121,13 @@ class NgxFetch : public PoolElement<NgxFetch> {
  AsyncFetch* async_fetch_;
  ResponseHeadersParser parser_;
  MessageHandler* message_handler_;
-  size_t bytes_received_;
+  int64 bytes_received_;
  int64 fetch_start_ms_;
  int64 fetch_end_ms_;
  int64 timeout_ms_;
  bool done_;
  int64 content_length_;
+  bool content_length_known_;

  struct sockaddr_in sin_;
  ngx_log_t* log_;
@@ -18,13 +18,13 @@

 #include <signal.h>

-#include "apr_time.h"
-
 #include "net/instaweb/util/public/abstract_mutex.h"
 #include "net/instaweb/util/public/debug.h"
 #include "net/instaweb/util/public/shared_circular_buffer.h"
 #include "net/instaweb/util/public/string_util.h"
 #include "net/instaweb/public/version.h"
+#include "pagespeed/kernel/base/posix_timer.h"
+#include "pagespeed/kernel/base/time_util.h"

 namespace {

@@ -118,10 +118,9 @@ void NgxMessageHandler::MessageVImpl(MessageType type, const char* msg,
  // Prepend time and severity to message.
  // Format is [time] [severity] [pid] message.
  GoogleString message;
-  char time_buffer[APR_CTIME_LEN + 1];
-  const char* time = time_buffer;
-  apr_status_t status = apr_ctime(time_buffer, apr_time_now());
-  if (status != APR_SUCCESS) {
+  GoogleString time;
+  PosixTimer timer;
+  if (!ConvertTimeToString(timer.NowMs(), &time)) {
    time = "?";
  }
  StrAppend(&message, "[", time, "] ",
@@ -149,14 +148,4 @@ void NgxMessageHandler::FileMessageVImpl(MessageType type, const char* file,
  }
 }

-// TODO(sligocki): It'd be nice not to do so much string copying.
-GoogleString NgxMessageHandler::Format(const char* msg, va_list args) {
-  GoogleString buffer;
-
-  // Ignore the name of this routine: it formats with vsnprintf.
-  // See base/stringprintf.cc.
-  StringAppendV(&buffer, msg, args);
-  return buffer;
-}
-
 }  // namespace net_instaweb
@@ -70,7 +70,6 @@ class NgxMessageHandler : public GoogleMessageHandler {

 private:
  ngx_uint_t GetNgxLogLevel(MessageType type);
-  GoogleString Format(const char* msg, va_list args);

  scoped_ptr<AbstractMutex> mutex_;
  GoogleString pid_string_;
@@ -36,27 +36,28 @@
 #include "ngx_rewrite_options.h"
 #include "ngx_server_context.h"

-#include "apr_time.h"
-
 #include "net/instaweb/automatic/public/proxy_fetch.h"
+#include "net/instaweb/http/public/async_fetch.h"
 #include "net/instaweb/http/public/cache_url_async_fetcher.h"
 #include "net/instaweb/http/public/content_type.h"
 #include "net/instaweb/http/public/request_context.h"
+#include "net/instaweb/public/global_constants.h"
+#include "net/instaweb/public/version.h"
 #include "net/instaweb/rewriter/public/experiment_matcher.h"
 #include "net/instaweb/rewriter/public/experiment_util.h"
 #include "net/instaweb/rewriter/public/process_context.h"
 #include "net/instaweb/rewriter/public/resource_fetch.h"
 #include "net/instaweb/rewriter/public/rewrite_driver.h"
+#include "net/instaweb/rewriter/public/rewrite_options.h"
+#include "net/instaweb/rewriter/public/rewrite_query.h"
 #include "net/instaweb/rewriter/public/rewrite_stats.h"
 #include "net/instaweb/rewriter/public/static_asset_manager.h"
-#include "net/instaweb/system/public/handlers.h"
 #include "net/instaweb/system/public/in_place_resource_recorder.h"
 #include "net/instaweb/system/public/system_caches.h"
 #include "net/instaweb/system/public/system_request_context.h"
 #include "net/instaweb/system/public/system_rewrite_options.h"
+#include "net/instaweb/system/public/system_server_context.h"
 #include "net/instaweb/system/public/system_thread_system.h"
-#include "net/instaweb/public/global_constants.h"
-#include "net/instaweb/public/version.h"
 #include "net/instaweb/util/public/fallback_property_page.h"
 #include "net/instaweb/util/public/google_message_handler.h"
 #include "net/instaweb/util/public/google_url.h"
@@ -69,8 +70,10 @@
 #include "net/instaweb/util/public/string_writer.h"
 #include "net/instaweb/util/public/time_util.h"
 #include "net/instaweb/util/stack_buffer.h"
-#include "pagespeed/kernel/thread/pthread_shared_mem.h"
+#include "pagespeed/kernel/base/posix_timer.h"
+#include "pagespeed/kernel/http/query_params.h"
 #include "pagespeed/kernel/html/html_keywords.h"
+#include "pagespeed/kernel/thread/pthread_shared_mem.h"

 extern ngx_module_t ngx_pagespeed;

@@ -239,11 +242,21 @@ void copy_response_headers_from_ngx(const ngx_http_request_t* r,

  headers->set_status_code(r->headers_out.status);

+  if (r->headers_out.location != NULL) {
+    headers->Add(HttpAttributes::kLocation,
+                 str_to_string_piece(r->headers_out.location->value));
+  }
+
  // Manually copy over the content type because it's not included in
  // request_->headers_out.headers.
  headers->Add(HttpAttributes::kContentType,
               str_to_string_piece(r->headers_out.content_type));

+  // When we don't have a date header, set one with the current time.
+  if (headers->Lookup1(HttpAttributes::kDate) == NULL) {
+    headers->SetDate(ngx_current_msec);
+  }
+
  // TODO(oschaaf): ComputeCaching should be called in setupforhtml()?
  headers->ComputeCaching();
 }
@@ -334,8 +347,6 @@ ngx_int_t copy_response_headers_to_ngx(
    // copy here?
    } else if (STR_EQ_LITERAL(name, "Connection")) {
      continue;
-    } else if (STR_EQ_LITERAL(name, "Vary")) {
-      continue;
    } else if (STR_EQ_LITERAL(name, "Keep-Alive")) {
      continue;
    } else if (STR_EQ_LITERAL(name, "Transfer-Encoding")) {
@@ -430,12 +441,22 @@ enum Response {
 };
 }  // namespace RequestRouting

+char* ps_main_configure(ngx_conf_t* cf, ngx_command_t* cmd, void* conf);
 char* ps_srv_configure(ngx_conf_t* cf, ngx_command_t* cmd, void* conf);
 char* ps_loc_configure(ngx_conf_t* cf, ngx_command_t* cmd, void* conf);

+// TODO(jud): Verify that all the offsets should be NGX_HTTP_SRV_CONF_OFFSET and
+// not NGX_HTTP_LOC_CONF_OFFSET or NGX_HTTP_MAIN_CONF_OFFSET.
 ngx_command_t ps_commands[] = {
  { ngx_string("pagespeed"),
-    NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1|
+    NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1|
+    NGX_CONF_TAKE2|NGX_CONF_TAKE3|NGX_CONF_TAKE4|NGX_CONF_TAKE5,
+    ps_main_configure,
+    NGX_HTTP_SRV_CONF_OFFSET,
+    0,
+    NULL },
+  { ngx_string("pagespeed"),
+    NGX_HTTP_SRV_CONF|NGX_CONF_TAKE1|
    NGX_CONF_TAKE2|NGX_CONF_TAKE3|NGX_CONF_TAKE4|NGX_CONF_TAKE5,
    ps_srv_configure,
    NGX_HTTP_SRV_CONF_OFFSET,
@@ -443,7 +464,7 @@ ngx_command_t ps_commands[] = {
    NULL },

  { ngx_string("pagespeed"),
-    NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1|
+    NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF|NGX_CONF_TAKE1|
    NGX_CONF_TAKE2|NGX_CONF_TAKE3|NGX_CONF_TAKE4|NGX_CONF_TAKE5,
    ps_loc_configure,
    NGX_HTTP_SRV_CONF_OFFSET,
@@ -462,80 +483,6 @@ void ps_ignore_sigpipe() {
  sigaction(SIGPIPE, &act, NULL);
 }

-namespace PsConfigure {
-enum OptionLevel {
-  kServer,
-  kLocation,
-};
-}  // namespace PsConfigure
-
-// These options are copied from mod_instaweb.cc, where
-// APACHE_CONFIG_OPTIONX indicates that they can not be set at the
-// directory/location level. They are not alphabetized on purpose,
-// but rather left in the same order as in mod_instaweb.cc in case
-// we end up needing te compare.
-// TODO(oschaaf): this duplication is a short term solution.
-const char* const global_only_options[] = {
-  "BlockingRewriteKey",
-  "CacheFlushFilename",
-  "CacheFlushPollIntervalSec",
-  "DangerPermitFetchFromUnknownHosts",
-  "CriticalImagesBeaconEnabled",
-  "ExperimentalFetchFromModSpdy",
-  "FetcherTimeoutMs",
-  "FetchHttps",
-  "FetchWithGzip",
-  "FileCacheCleanIntervalMs",
-  "FileCacheInodeLimit",
-  "FileCachePath",
-  "FileCacheSizeKb",
-  "ForceCaching",
-  "ImageMaxRewritesAtOnce",
-  "ImgMaxRewritesAtOnce",
-  "InheritVHostConfig",
-  "InstallCrashHandler",
-  "LRUCacheByteLimit",
-  "LRUCacheKbPerProcess",
-  "MaxCacheableContentLength",
-  "MemcachedServers",
-  "MemcachedThreads",
-  "MemcachedTimeoutUs",
-  "MessageBufferSize",
-  "NumRewriteThreads",
-  "NumExpensiveRewriteThreads",
-  "RateLimitBackgroundFetches",
-  "ReportUnloadTime",
-  "RespectXForwardedProto",
-  "SharedMemoryLocks",
-  "SlurpDirectory",
-  "SlurpFlushLimit",
-  "SlurpReadOnly",
-  "SupportNoScriptEnabled",
-  "StatisticsLoggingChartsCSS",
-  "StatisticsLoggingChartsJS",
-  "TestProxy",
-  "TestProxySlurp",
-  "TrackOriginalContentLength",
-  "UsePerVHostStatistics",
-  "XHeaderValue",
-  "LoadFromFile",
-  "LoadFromFileMatch",
-  "LoadFromFileRule",
-  "LoadFromFileRuleMatch",
-  "UseNativeFetcher"
-};
-
-bool ps_is_global_only_option(const StringPiece& option_name) {
-  ngx_uint_t i;
-  ngx_uint_t size = sizeof(global_only_options) / sizeof(char*);
-  for (i = 0; i < size; i++) {
-    if (StringCaseEqual(global_only_options[i], option_name)) {
-      return true;
-    }
-  }
-  return false;
-}
-
 char* ps_init_dir(const StringPiece& directive,
                  const StringPiece& path,
                  ngx_conf_t* cf) {
@@ -563,15 +510,24 @@ char* ps_init_dir(const StringPiece& directive,
    return NULL;  // We're not root, so we're staying whoever we are.
  }

-  ngx_core_conf_t* ccf =
-      (ngx_core_conf_t*)(ngx_get_conf(cf->cycle->conf_ctx, ngx_core_module));
+  // chown if owner differs from nginx worker user.
+  ngx_core_conf_t* ccf = reinterpret_cast<ngx_core_conf_t*>(
+      ngx_get_conf(cf->cycle->conf_ctx, ngx_core_module));
  CHECK(ccf != NULL);
-
-  if (chown(gs_path.c_str(), ccf->user, ccf->group) != 0) {
+  struct stat gs_stat;
+  if (stat(gs_path.c_str(), &gs_stat) != 0) {
    return string_piece_to_pool_string(
        cf->pool, net_instaweb::StrCat(
-            directive, " ", path, " unable to set permissions"));
+            directive, " ", path, " stat() failed"));
  }
+  if (gs_stat.st_uid != ccf->user) {
+    if (chown(gs_path.c_str(), ccf->user, ccf->group) != 0) {
+      return string_piece_to_pool_string(
+          cf->pool, net_instaweb::StrCat(
+              directive, " ", path, " unable to set permissions"));
+    }
+  }
+
  return NULL;
 }

@@ -579,7 +535,7 @@ char* ps_init_dir(const StringPiece& directive,
 char* ps_configure(ngx_conf_t* cf,
                   NgxRewriteOptions** options,
                   MessageHandler* handler,
-                   PsConfigure::OptionLevel option_level) {
+                   net_instaweb::RewriteOptions::OptionScope option_scope) {
  // args[0] is always "pagespeed"; ignore it.
  ngx_uint_t n_args = cf->args->nelts - 1;

@@ -594,19 +550,6 @@ char* ps_configure(ngx_conf_t* cf,
    args[i] = str_to_string_piece(value[i+1]);
  }

-  if (StringCaseEqual("UseNativeFetcher", args[0])) {
-    if (option_level != PsConfigure::kServer) {
-      return const_cast<char*>(
-          "UseNativeFetcher can only be set in the http{} block.");
-    }
-  }
-  if (option_level == PsConfigure::kLocation && n_args > 1) {
-    if (ps_is_global_only_option(args[0])) {
-      return string_piece_to_pool_string(cf->pool, StrCat(
-          "\"", args[0], "\" cannot be set at location scope"));
-    }
-  }
-
  // Some options require the worker process to be able to read and write to
  // a specific directory.  Generally the master process is root while the
  // worker is nobody, so we need to change permissions and create the directory
@@ -629,25 +572,33 @@ char* ps_configure(ngx_conf_t* cf,
        cfg_m->driver_factory->thread_system());
  }
  const char* status = (*options)->ParseAndSetOptions(
-      args, n_args, cf->pool, handler, cfg_m->driver_factory);
+      args, n_args, cf->pool, handler, cfg_m->driver_factory, option_scope);

  // nginx expects us to return a string literal but doesn't mark it const.
  return const_cast<char*>(status);
 }

+char* ps_main_configure(ngx_conf_t* cf, ngx_command_t* cmd, void* conf) {
+  ps_srv_conf_t* cfg_s = static_cast<ps_srv_conf_t*>(
+      ngx_http_conf_get_module_srv_conf(cf, ngx_pagespeed));
+  return ps_configure(cf, &cfg_s->options, cfg_s->handler,
+                      net_instaweb::RewriteOptions::kProcessScopeStrict);
+}
+
 char* ps_srv_configure(ngx_conf_t* cf, ngx_command_t* cmd, void* conf) {
  ps_srv_conf_t* cfg_s = static_cast<ps_srv_conf_t*>(
      ngx_http_conf_get_module_srv_conf(cf, ngx_pagespeed));
  return ps_configure(cf, &cfg_s->options, cfg_s->handler,
-                      PsConfigure::kServer);
+                      net_instaweb::RewriteOptions::kServerScope);
 }

 char* ps_loc_configure(ngx_conf_t* cf, ngx_command_t* cmd, void* conf) {
  ps_loc_conf_t* cfg_l = static_cast<ps_loc_conf_t*>(
      ngx_http_conf_get_module_loc_conf(cf, ngx_pagespeed));

-  return ps_configure(cf, &cfg_l->options, cfg_l->handler,
-                      PsConfigure::kLocation);
+  return ps_configure(
+      cf, &cfg_l->options, cfg_l->handler,
+      net_instaweb::RewriteOptions::kDirectoryScope);
 }

 void ps_cleanup_loc_conf(void* data) {
@@ -734,6 +685,7 @@ void* ps_create_main_conf(ngx_conf_t* cf) {
  NgxRewriteDriverFactory::Initialize();

  cfg_m->driver_factory = new NgxRewriteDriverFactory(
+      *process_context,
      new SystemThreadSystem(),
      "" /* hostname, not used */,
      -1 /* port, not used */);
@@ -848,19 +800,28 @@ char* ps_merge_srv_conf(ngx_conf_t* cf, void* parent, void* child) {
 }

 char* ps_merge_loc_conf(ngx_conf_t* cf, void* parent, void* child) {
-  ps_loc_conf_t* parent_cfg_l = static_cast<ps_loc_conf_t*>(parent);
-
-  // The variant of the pagespeed directive that is acceptable in location
-  // blocks is only acceptable in location blocks, so we should never be merging
-  // in options from a server or main block.
-  CHECK(parent_cfg_l->options == NULL);
-
  ps_loc_conf_t* cfg_l = static_cast<ps_loc_conf_t*>(child);
  if (cfg_l->options == NULL) {
    // No directory specific options.
    return NGX_CONF_OK;
  }

+  // While you can't put a "location" block inside a "location" block you can
+  // put an "if" block inside a "location" block, which is implemented by making
+  // a pretend "location" block.  In this case we may have pagespeed options
+  // from the parent "location" block as well as from the current locationish
+  // "if" block.
+  ps_loc_conf_t* parent_cfg_l = static_cast<ps_loc_conf_t*>(parent);
+  if (parent_cfg_l->options != NULL) {
+    // Rebase our options off of the ones defined in the parent location block.
+    ps_merge_options(parent_cfg_l->options, &cfg_l->options);
+    return NGX_CONF_OK;
+  }
+
+  // Pagespeed options are defined in this location block, and it either has no
+  // parent (typical case) or is an if block whose parent location block defines
+  // no pagespeed options.  Base our options off of those in the server block.
+
  ps_srv_conf_t* cfg_s = static_cast<ps_srv_conf_t*>(
      ngx_http_conf_get_module_srv_conf(cf, ngx_pagespeed));

@@ -959,18 +920,9 @@ int ps_determine_port(ngx_http_request_t* r) {

  return port;
 }
+}  // namespace

-GoogleString ps_determine_url(ngx_http_request_t* r) {
-  int port = ps_determine_port(r);
-  GoogleString port_string;
-  if ((ps_is_https(r) && (port == 443 || port == -1)) ||
-      (!ps_is_https(r) && (port == 80 || port == -1))) {
-    // No port specifier needed for requests on default ports.
-    port_string = "";
-  } else {
-    port_string = StrCat(":", IntegerToString(port));
-  }
-
+StringPiece ps_determine_host(ngx_http_request_t* r) {
  StringPiece host = str_to_string_piece(r->headers_in.server);
  if (host.size() == 0) {
    // If host is unspecified, perhaps because of a pure HTTP 1.0 "GET /path",
@@ -985,6 +937,23 @@ GoogleString ps_determine_url(ngx_http_request_t* r) {
    }
    host = str_to_string_piece(s);
  }
+  return host;
+}
+
+namespace {
+
+GoogleString ps_determine_url(ngx_http_request_t* r) {
+  int port = ps_determine_port(r);
+  GoogleString port_string;
+  if ((ps_is_https(r) && (port == 443 || port == -1)) ||
+      (!ps_is_https(r) && (port == 80 || port == -1))) {
+    // No port specifier needed for requests on default ports.
+    port_string = "";
+  } else {
+    port_string = StrCat(":", IntegerToString(port));
+  }
+
+  StringPiece host = ps_determine_host(r);

  return StrCat(ps_is_https(r) ? "https://" : "http://",
                host, port_string, str_to_string_piece(r->unparsed_uri));
@@ -1265,11 +1234,9 @@ RewriteOptions* ps_determine_request_options(
  // make cache key consistent for both lookup and storing in cache.
  //
  // Sets option from request headers and url.
-  ServerContext::OptionsBoolPair query_options_success =
-      cfg_s->server_context->GetQueryOptions(url, request_headers,
-                                             response_headers);
-  bool get_query_options_success = query_options_success.second;
-  if (!get_query_options_success) {
+  RewriteQuery rewrite_query;
+  if (!cfg_s->server_context->GetQueryOptions(
+          url, request_headers, response_headers, &rewrite_query)) {
    // Failed to parse query params or request headers.  Treat this as if there
    // were no query params given.
    ngx_log_error(NGX_LOG_INFO, r->connection->log, 0,
@@ -1279,7 +1246,7 @@ RewriteOptions* ps_determine_request_options(

  // Will be NULL if there aren't any options set with query params or in
  // headers.
-  return query_options_success.first;
+  return rewrite_query.ReleaseOptions();
 }

 // Check whether this visitor is already in an experiment.  If they're not,
@@ -1296,8 +1263,9 @@ bool ps_set_experiment_state_and_cookie(ngx_http_request_t* r,
  bool need_cookie = cfg_s->server_context->experiment_matcher()->
      ClassifyIntoExperiment(*request_headers, options);
  if (need_cookie && host.length() > 0) {
-    int64 time_now_us = apr_time_now();
-    int64 expiration_time_ms = (time_now_us/1000 +
+    PosixTimer timer;
+    int64 time_now_ms = timer.NowMs();
+    int64 expiration_time_ms = (time_now_ms +
                                options->experiment_cookie_duration_ms());

    // TODO(jefftk): refactor SetExperimentCookie to expose the value we want to
@@ -1341,7 +1309,8 @@ bool ps_determine_options(ngx_http_request_t* r,
                          RequestHeaders* request_headers,
                          ResponseHeaders* response_headers,
                          RewriteOptions** options,
-                          GoogleUrl* url) {
+                          GoogleUrl* url,
+                          bool html_rewrite) {
  ps_srv_conf_t* cfg_s = ps_get_srv_config(r);
  ps_loc_conf_t* cfg_l = ps_get_loc_config(r);

@@ -1356,11 +1325,12 @@ bool ps_determine_options(ngx_http_request_t* r,
  // rebased on the directory options or the global options.
  RewriteOptions* request_options = ps_determine_request_options(
      r, request_headers, response_headers, cfg_s, url);
+  bool have_request_options = request_options != NULL;

  // Because the caller takes ownership of any options we return, the only
  // situation in which we can avoid allocating a new RewriteOptions is if the
  // global options are ok as are.
-  if (directory_options == NULL && request_options == NULL &&
+  if (!have_request_options && directory_options == NULL &&
      !global_options->running_experiment()) {
    return true;
  }
@@ -1372,14 +1342,24 @@ bool ps_determine_options(ngx_http_request_t* r,
    *options = global_options->Clone();
  }

-  // Modify our options in response to request options or experiment settings,
-  // if we need to.  If there are request options then ignore the experiment
-  // because we don't want experiments to be contaminated with unexpected
-  // settings.
-  if (request_options != NULL) {
+  // Modify our options in response to request options if specified.
+  if (have_request_options) {
    (*options)->Merge(*request_options);
    delete request_options;
-  } else if ((*options)->running_experiment()) {
+    request_options = NULL;
+  }
+
+  // If we're running an experiment and processing html then modify our options
+  // in response to the experiment.  Except we generally don't want experiments
+  // to be contaminated with unexpected settings, so ignore experiments if we
+  // have request-specific options.  Unless EnrollExperiment is on, probably set
+  // by a query parameter, in which case we want to go ahead and apply the
+  // experimental settings even if it means bad data, because we're just seeing
+  // what it looks like.
+  if ((*options)->running_experiment() &&
+      html_rewrite &&
+      (!have_request_options ||
+       (*options)->enroll_experiment())) {
    bool ok = ps_set_experiment_state_and_cookie(
        r, request_headers, *options, url->Host());
    if (!ok) {
@@ -1416,7 +1396,8 @@ bool ps_apply_x_forwarded_proto(ngx_http_request_t* r, GoogleString* url) {
    return false;  // No X-Forwarded-Proto header found.
  }

-  StringPiece x_forwarded_proto = str_to_string_piece(*x_forwarded_proto_header);
+  StringPiece x_forwarded_proto =
+      str_to_string_piece(*x_forwarded_proto_header);
  if (!STR_CASE_EQ_LITERAL(*x_forwarded_proto_header, "http") &&
      !STR_CASE_EQ_LITERAL(*x_forwarded_proto_header, "https")) {
    LOG(WARNING) << "Unsupported X-Forwarded-Proto: " << x_forwarded_proto
@@ -1542,11 +1523,6 @@ void ps_release_request_context(void* data) {
    ctx->recorder = NULL;
  }

-  if (ctx->ipro_response_headers != NULL) {
-    delete ctx->ipro_response_headers;
-    ctx->ipro_response_headers = NULL;
-  }
-
  ps_release_base_fetch(ctx);
  delete ctx;
 }
@@ -1579,7 +1555,8 @@ RequestRouting::Response ps_route_request(ngx_http_request_t* r,

  if (is_pagespeed_subrequest(r)) {
    return RequestRouting::kPagespeedSubrequest;
-  } else if (url.PathSansLeaf() == NgxRewriteDriverFactory::kStaticAssetPrefix) {
+  } else if (url.PathSansLeaf() ==
+             NgxRewriteDriverFactory::kStaticAssetPrefix) {
    return RequestRouting::kStaticContent;
  } else if (url.PathSansQuery() == "/ngx_pagespeed_statistics" ||
             url.PathSansQuery() == "/ngx_pagespeed_global_statistics" ) {
@@ -1616,7 +1593,9 @@ ngx_int_t ps_resource_handler(ngx_http_request_t* r, bool html_rewrite) {

  CHECK(!(html_rewrite && (ctx == NULL || ctx->html_rewrite == false)));

-  if (!html_rewrite && r->method != NGX_HTTP_GET && r->method != NGX_HTTP_HEAD) {
+  if (!html_rewrite &&
+      r->method != NGX_HTTP_GET &&
+      r->method != NGX_HTTP_HEAD) {
    return NGX_DECLINED;
  }

@@ -1634,7 +1613,7 @@ ngx_int_t ps_resource_handler(ngx_http_request_t* r, bool html_rewrite) {
  RewriteOptions* options = NULL;

  if (!ps_determine_options(r, request_headers.get(), response_headers.get(),
-                            &options, &url)) {
+                            &options, &url, html_rewrite)) {
    return NGX_ERROR;
  }

@@ -1653,7 +1632,7 @@ ngx_int_t ps_resource_handler(ngx_http_request_t* r, bool html_rewrite) {
  // parameters.  Keep url_string in sync with url.
  url.Spec().CopyToString(&url_string);

-  if (options->respect_x_forwarded_proto()) {
+  if (cfg_s->server_context->global_options()->respect_x_forwarded_proto()) {
    bool modified_url = ps_apply_x_forwarded_proto(r, &url_string);
    if (modified_url) {
      url.Reset(url_string);
@@ -1671,20 +1650,33 @@ ngx_int_t ps_resource_handler(ngx_http_request_t* r, bool html_rewrite) {
    ctx = new ps_request_ctx_t();

    ctx->r = r;
-    ctx->ipro_response_headers = NULL;
    ctx->write_pending = false;
    ctx->html_rewrite = false;
    ctx->in_place = false;
    ctx->pagespeed_connection = NULL;
    // See build_context_for_request() in mod_instaweb.cc
+    // TODO(jefftk): Is this the right place to be modifying caching headers for
+    // html fetches?  Or should that be done later, in the headers flow for
+    // filter mode, rather than here in resource fetch mode?
    if (!options->modify_caching_headers()) {
      ctx->preserve_caching_headers = kPreserveAllCachingHeaders;
-    } else if (!options->downstream_cache_purge_location_prefix().empty()) {
-      ctx->preserve_caching_headers = kPreserveOnlyCacheControl;
-    } else {
+    } else if (!options->IsDownstreamCacheIntegrationEnabled()) {
+      // Downstream cache integration is not enabled. Disable original
+      // Cache-Control headers.
      ctx->preserve_caching_headers = kDontPreserveHeaders;
+    } else {
+      ctx->preserve_caching_headers = kPreserveOnlyCacheControl;
+      // Downstream cache integration is enabled. If a rebeaconing key has been
+      // configured and there is a ShouldBeacon header with the correct key,
+      // disable original Cache-Control headers so that the instrumented page is
+      // served out with no-cache.
+      StringPiece should_beacon(request_headers->Lookup1(kPsaShouldBeacon));
+      if (options->MatchesDownstreamCacheRebeaconingKey(should_beacon)) {
+        ctx->preserve_caching_headers = kDontPreserveHeaders;
+      }
    }
    ctx->recorder = NULL;
+    ctx->url_string = url_string;

    // Set up a cleanup handler on the request.
    ngx_http_cleanup_t* cleanup = ngx_http_cleanup_add(r, 0);
@@ -2132,12 +2124,40 @@ ngx_http_output_body_filter_pt ngx_http_next_body_filter;
 ngx_int_t ps_in_place_check_header_filter(ngx_http_request_t* r) {
  ps_request_ctx_t* ctx = ps_get_request_context(r);

-  if (ctx == NULL || !ctx->in_place) {
+  if (ctx == NULL) {
+    return ngx_http_next_header_filter(r);
+  }
+
+  if (ctx->recorder != NULL) {
+    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                   "ps in place check header filter recording: %V", &r->uri);
+
+    CHECK(!ctx->in_place);
+
+    // We didn't find this resource in cache originally, so we're recording it
+    // as it passes us by.  At this point the headers from things that run
+    // before us are set but not things that run after us, which means here is
+    // where we need to check whether there's a "Content-Encoding: gzip".  If we
+    // waited to do this in ps_in_place_body_filter we wouldn't be able to tell
+    // the difference between response headers that have "C-E: gz" because we're
+    // proxying for an upstream that gzipped the content and response headers
+    // that have it because the gzip filter (which runs after us) is going to
+    // produce gzipped output.
+    //
+    // The recorder will do this checking, so pass it the headers.
+    ResponseHeaders response_headers;
+    copy_response_headers_from_ngx(r, &response_headers);
+    ctx->recorder->ConsiderResponseHeaders(
+        InPlaceResourceRecorder::kPreliminaryHeaders, &response_headers);
+    return ngx_http_next_header_filter(r);
+  }
+
+  if (!ctx->in_place) {
    return ngx_http_next_header_filter(r);
  }

  ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
-                 "ps in place check header filter: %V", &r->uri);
+                 "ps in place check header filter initial: %V", &r->uri);

  int status_code = r->headers_out.status;
  bool status_ok = (status_code != 0) && (status_code < 400);
@@ -2146,6 +2166,9 @@ ngx_int_t ps_in_place_check_header_filter(ngx_http_request_t* r) {
  NgxServerContext* server_context = cfg_s->server_context;
  MessageHandler* message_handler = cfg_s->handler;
  GoogleString url = ps_determine_url(r);
+  // The URL we use for cache key is a bit different since it may
+  // have PageSpeed query params removed.
+  GoogleString cache_url = ctx->url_string;

  // continue process
  if (status_ok) {
@@ -2159,32 +2182,39 @@ ngx_int_t ps_in_place_check_header_filter(ngx_http_request_t* r) {
    return ngx_http_next_header_filter(r);
  }

-  if (status_code == CacheUrlAsyncFetcher::kNotInCacheStatus) {
+  if (status_code == CacheUrlAsyncFetcher::kNotInCacheStatus &&
+      !r->header_only) {
    server_context->rewrite_stats()->ipro_not_in_cache()->Add(1);
    server_context->message_handler()->Message(
        kInfo,
        "Could not rewrite resource in-place "
        "because URL is not in cache: %s",
-        url.c_str());
+        cache_url.c_str());
    const SystemRewriteOptions* options = SystemRewriteOptions::DynamicCast(
        ctx->driver->options());
-    scoped_ptr<RequestHeaders> request_headers(new RequestHeaders);
-    copy_request_headers_from_ngx(r, request_headers.get());
+    RequestHeaders request_headers;
+    copy_request_headers_from_ngx(r, &request_headers);
    // This URL was not found in cache (neither the input resource nor
    // a ResourceNotCacheable entry) so we need to get it into cache
    // (or at least a note that it cannot be cached stored there).
    // We do that using an Apache output filter.
    ctx->recorder = new InPlaceResourceRecorder(
-        url,
-        request_headers.release(),
+        RequestContextPtr(cfg_s->server_context->NewRequestContext(r)),
+        cache_url,
+        ctx->driver->CacheFragment(),
+        request_headers.GetProperties(),
        options->respect_vary(),
        options->ipro_max_response_bytes(),
        options->ipro_max_concurrent_recordings(),
+        options->implicit_cache_ttl_ms(),
        server_context->http_cache(),
        server_context->statistics(),
        message_handler);
    // set in memory flag for in place_body_filter
    r->filter_need_in_memory = 1;
+
+    // We don't have the response headers at all yet because we haven't yet gone
+    // to the backend.
  } else {
    server_context->rewrite_stats()->ipro_not_rewritable()->Add(1);
    message_handler->Message(kInfo,
@@ -2210,42 +2240,11 @@ ngx_int_t ps_in_place_body_filter(ngx_http_request_t* r, ngx_chain_t* in) {
                 "ps in place body filter: %V", &r->uri);

  InPlaceResourceRecorder* recorder = ctx->recorder;
-
-  if (ctx->ipro_response_headers == NULL) {
-    // Prepare response headers.
-    ctx->ipro_response_headers = new ResponseHeaders();
-
-    // TODO(oschaaf): We don't get a Date response header here.
-    // Currently, we invent one and set it to the current date/time.
-    // We need to investigate why we don't receive it.
-    ctx->ipro_response_headers->set_major_version(r->http_version / 1000);
-    ctx->ipro_response_headers->set_minor_version(r->http_version % 1000);
-    copy_headers_from_table(r->headers_out.headers, ctx->ipro_response_headers);
-    ctx->ipro_response_headers->set_status_code(r->headers_out.status);
-    ctx->ipro_response_headers->Add(HttpAttributes::kContentType,
-                str_to_string_piece(r->headers_out.content_type));
-    if (r->headers_out.location != NULL) {
-      ctx->ipro_response_headers->Add(HttpAttributes::kLocation,
-                  str_to_string_piece(r->headers_out.location->value));
-    }
-    StringPiece date =
-        ctx->ipro_response_headers->Lookup1(HttpAttributes::kDate);
-    if (date.empty()) {
-      ctx->ipro_response_headers->SetDate(ngx_current_msec);
-    }
-    ctx->ipro_response_headers->ComputeCaching();
-
-    // Unlike in Apache we get the final response headers before we get the
-    // content.  This means we can consider them earlier and abort the
-    // request if need be without buffering everything.
-    recorder->ConsiderResponseHeaders(ctx->ipro_response_headers);
-  }
-
  for (ngx_chain_t* cl = in; cl; cl = cl->next) {
    if (ngx_buf_size(cl->buf)) {
       CHECK(ngx_buf_in_memory(cl->buf));
-       StringPiece contents(reinterpret_cast<char *>(cl->buf->pos),
-                                 ngx_buf_size(cl->buf));
+       StringPiece contents(reinterpret_cast<char*>(cl->buf->pos),
+                            ngx_buf_size(cl->buf));
       recorder->Write(contents, recorder->handler());
    }

@@ -2254,7 +2253,9 @@ ngx_int_t ps_in_place_body_filter(ngx_http_request_t* r, ngx_chain_t* in) {
    }

    if (cl->buf->last_buf || recorder->failed()) {
-      ctx->recorder->DoneAndSetHeaders(ctx->ipro_response_headers);
+      ResponseHeaders response_headers;
+      copy_response_headers_from_ngx(r, &response_headers);
+      ctx->recorder->DoneAndSetHeaders(&response_headers);
      ctx->recorder = NULL;
      break;
    }
@@ -2304,6 +2305,27 @@ ngx_int_t send_out_headers_and_body(
  return ngx_http_output_filter(r, out);
 }

+namespace {
+
+// TODO(jefftk): This class is a temporary shim to just support console fetch,
+// but we eventually want to convert everything in ps_simple_handler to use
+// something akin to NgxBaseFetch (which sends results direct to nginx).  This
+// is work in progress (but well along) in the mod_pagespeed world.
+class NgxPagespeedConsoleAsyncFetch : public AsyncFetchUsingWriter {
+ public:
+  NgxPagespeedConsoleAsyncFetch(const RequestContextPtr& request_context,
+                                Writer* writer)
+      : AsyncFetchUsingWriter(request_context, writer) { }
+  virtual void HandleDone(bool status) { }
+  virtual void HandleHeadersComplete() { }
+
+  void FlushToNgx(const GoogleString& output, ngx_http_request_t* r) {
+    send_out_headers_and_body(r, *response_headers(), output);
+  }
+};
+
+}  // namespace
+
 ngx_int_t ps_simple_handler(ngx_http_request_t* r,
                            NgxServerContext* server_context,
                            RequestRouting::Response response_category) {
@@ -2313,6 +2335,13 @@ ngx_int_t ps_simple_handler(ngx_http_request_t* r,
  NgxMessageHandler* message_handler = factory->ngx_message_handler();
  StringPiece request_uri_path = str_to_string_piece(r->uri);

+  GoogleString url_string = ps_determine_url(r);
+  GoogleUrl url(url_string);
+  QueryParams query_params;
+  if (url.IsWebValid()) {
+    query_params.Parse(url.Query());
+  }
+
  GoogleString output;
  StringWriter writer(&output);
  HttpStatus::Code status = HttpStatus::kOK;
@@ -2332,22 +2361,33 @@ ngx_int_t ps_simple_handler(ngx_http_request_t* r,
      file_contents.CopyToString(&output);
      break;
    }
-    case RequestRouting::kStatistics:
-      error_message = StatisticsHandler(
-          factory,
-          server_context,
-          NULL,  // No SPDY-specific config in ngx_pagespeed.
-          !factory->use_per_vhost_statistics() || StringCaseStartsWith(
-              request_uri_path, "/ngx_pagespeed_global_statistics"),
-          StringPiece(reinterpret_cast<char*>(r->args.data), r->args.len),
-          &content_type,
-          &writer,
-          message_handler);
-      break;
-    case RequestRouting::kConsole:
-      ConsoleHandler(
-          server_context, server_context->config(), &writer, message_handler);
-      break;
+    case RequestRouting::kStatistics: {
+      bool is_global_request =
+          StringCaseStartsWith(
+              request_uri_path, "/ngx_pagespeed_global_statistics");
+      ps_srv_conf_t* cfg_s = ps_get_srv_config(r);
+      RequestContextPtr request_context(
+          cfg_s->server_context->NewRequestContext(r));
+      NgxPagespeedConsoleAsyncFetch fetch(request_context, &writer);
+      server_context->StatisticsPage(
+          is_global_request,
+          query_params,
+          &fetch);
+      fetch.FlushToNgx(output, r);
+      return NGX_OK;
+    }
+    case RequestRouting::kConsole: {
+      ps_srv_conf_t* cfg_s = ps_get_srv_config(r);
+      RequestContextPtr request_context(
+          cfg_s->server_context->NewRequestContext(r));
+      NgxPagespeedConsoleAsyncFetch fetch(request_context, &writer);
+      server_context->ConsoleHandler(*server_context->config(),
+                                     SystemServerContext::kStatistics,
+                                     query_params,
+                                     &fetch);
+      fetch.FlushToNgx(output, r);
+      return NGX_OK;
+    }
    case RequestRouting::kMessages: {
      GoogleString log;
      StringWriter log_writer(&log);
@@ -2392,8 +2432,7 @@ ngx_int_t ps_simple_handler(ngx_http_request_t* r,

  char* cache_control_s = string_piece_to_pool_string(r->pool, cache_control);
  if (cache_control_s != NULL) {
-    if (FindIgnoreCase(cache_control, "private") ==
-        static_cast<int>(StringPiece::npos)) {
+    if (FindIgnoreCase(cache_control, "private") == StringPiece::npos) {
      response_headers.Add(HttpAttributes::kEtag, "W/\"0\"");
    }
  }
@@ -2843,6 +2882,8 @@ ngx_int_t ps_init_child_process(ngx_cycle_t* cycle) {
    return NGX_OK;
  }

+  SystemRewriteDriverFactory::InitApr();
+
  // ChildInit() will initialise all ServerContexts, which we need to
  // create ProxyFetchFactories below
  cfg_m->driver_factory->LoggingInit(cycle->log);
@@ -103,6 +103,10 @@ typedef struct {
  RewriteDriver* driver;
  InPlaceResourceRecorder* recorder;
  ResponseHeaders* ipro_response_headers;
+
+  // We need to remember the URL here as well since we may modify what NGX
+  // gets by stripping our special query params and honoring X-Forwarded-Proto.
+  GoogleString url_string;
 } ps_request_ctx_t;


@@ -117,6 +121,8 @@ ngx_int_t copy_response_headers_to_ngx(
    const ResponseHeaders& pagespeed_headers,
    PreserveCachingHeaders preserve_caching_headers);

+StringPiece ps_determine_host(ngx_http_request_t* r);
+
 }  // namespace net_instaweb

 #endif  // NGX_PAGESPEED_H_
@@ -69,8 +69,9 @@ const char NgxRewriteDriverFactory::kStaticAssetPrefix[] =
 class SharedCircularBuffer;

 NgxRewriteDriverFactory::NgxRewriteDriverFactory(
+    const ProcessContext& process_context,
    SystemThreadSystem* system_thread_system, StringPiece hostname, int port)
-    : SystemRewriteDriverFactory(system_thread_system,
+    : SystemRewriteDriverFactory(process_context, system_thread_system,
        NULL /* default shared memory runtime */, hostname, port),
      main_conf_(NULL),
      threads_started_(false),
@@ -82,7 +83,9 @@ NgxRewriteDriverFactory::NgxRewriteDriverFactory(
      log_(NULL),
      resolver_timeout_(NGX_CONF_UNSET_MSEC),
      use_native_fetcher_(false),
-      ngx_shared_circular_buffer_(NULL) {
+      ngx_shared_circular_buffer_(NULL),
+      hostname_(hostname.as_string()),
+      port_(port) {
  InitializeDefaultOptions();
  default_options()->set_beacon_url("/ngx_pagespeed_beacon");
  SystemRewriteOptions* system_options = dynamic_cast<SystemRewriteOptions*>(
@@ -177,6 +180,12 @@ NgxServerContext* NgxRewriteDriverFactory::MakeNgxServerContext(
  return server_context;
 }

+ServerContext* NgxRewriteDriverFactory::NewDecodingServerContext() {
+  ServerContext* sc = new NgxServerContext(this, hostname_, port_);
+  InitStubDecodingServerContext(sc);
+  return sc;
+}
+
 ServerContext* NgxRewriteDriverFactory::NewServerContext() {
  LOG(DFATAL) << "MakeNgxServerContext should be used instead";
  return NULL;
@@ -218,6 +227,8 @@ void NgxRewriteDriverFactory::LoggingInit(ngx_log_t* log) {
 void NgxRewriteDriverFactory::SetCircularBuffer(
    SharedCircularBuffer* buffer) {
  ngx_shared_circular_buffer_ = buffer;
+  ngx_message_handler_->set_buffer(buffer);
+  ngx_html_parse_message_handler_->set_buffer(buffer);
 }

 void NgxRewriteDriverFactory::SetServerContextMessageHandler(
@@ -28,7 +28,6 @@ extern "C" {

 #include <set>

-#include "apr_pools.h"
 #include "net/instaweb/system/public/system_rewrite_driver_factory.h"
 #include "net/instaweb/util/public/md5_hasher.h"
 #include "net/instaweb/util/public/scoped_ptr.h"
@@ -56,6 +55,7 @@ class NgxRewriteDriverFactory : public SystemRewriteDriverFactory {

  // We take ownership of the thread system.
  explicit NgxRewriteDriverFactory(
+      const ProcessContext& process_context,
      SystemThreadSystem* system_thread_system, StringPiece hostname, int port);
  virtual ~NgxRewriteDriverFactory();
  virtual Hasher* NewHasher();
@@ -71,6 +71,7 @@ class NgxRewriteDriverFactory : public SystemRewriteDriverFactory {
  // Initializes the StaticAssetManager.
  virtual void InitStaticAssetManager(
      StaticAssetManager* static_asset_manager);
+  virtual ServerContext* NewDecodingServerContext();
  bool InitNgxUrlAsyncFetchers();
  // Check resolver configured or not.
  bool CheckResolver();
@@ -80,7 +81,7 @@ class NgxRewriteDriverFactory : public SystemRewriteDriverFactory {
  // platform-independent statistics.
  static void InitStats(Statistics* statistics);
  NgxServerContext* MakeNgxServerContext(StringPiece hostname, int port);
-  ServerContext* NewServerContext();
+  virtual ServerContext* NewServerContext();

  // Starts pagespeed threads if they've not been started already.  Must be
  // called after the caller has finished any forking it intends to do.
@@ -169,6 +170,9 @@ class NgxRewriteDriverFactory : public SystemRewriteDriverFactory {
  // TODO(jefftk): merge the nginx and apache ways of doing this.
  SharedCircularBuffer* ngx_shared_circular_buffer_;

+  GoogleString hostname_;
+  int port_;
+
  DISALLOW_COPY_AND_ASSIGN(NgxRewriteDriverFactory);
 };

@@ -37,7 +37,38 @@ namespace net_instaweb {

 namespace {

-const char kNgxPagespeedStatisticsHandlerPath[] = "/ngx_pagespeed_statistics";
+// These options are copied from mod_instaweb.cc, where APACHE_CONFIG_OPTIONX
+// indicates that they can not be set at the directory/location level. They set
+// options in the RewriteDriverFactory, so they do not appear in RewriteOptions.
+// They are not alphabetized on purpose, but rather left in the same order as in
+// mod_instaweb.cc in case we end up needing to compare.
+// TODO(oschaaf): this duplication is a short term solution.
+const char* const server_only_options[] = {
+  "FetcherTimeoutMs",
+  "FetchProxy",
+  "ForceCaching",
+  "GeneratedFilePrefix",
+  "ImgMaxRewritesAtOnce",
+  "InheritVHostConfig",
+  "InstallCrashHandler",
+  "MessageBufferSize",
+  "NumRewriteThreads",
+  "NumExpensiveRewriteThreads",
+  "TrackOriginalContentLength",
+  "UsePerVHostStatistics",  // TODO(anupama): What to do about "No longer used"
+  "BlockingRewriteRefererUrls",
+  "CreateSharedMemoryMetadataCache",
+  "LoadFromFile",
+  "LoadFromFileMatch",
+  "LoadFromFileRule",
+  "LoadFromFileRuleMatch",
+  "UseNativeFetcher"
+};
+
+// Options that can only be used in the main (http) option scope.
+const char* const main_only_options[] = {
+  "UseNativeFetcher"
+};

 }  // namespace

@@ -58,11 +89,6 @@ void NgxRewriteOptions::Init() {
  DCHECK(ngx_properties_ != NULL)
      << "Call NgxRewriteOptions::Initialize() before construction";
  InitializeOptions(ngx_properties_);
-
-  // Nginx-specific default.
-  // TODO(sligocki): Get rid of this line and let both Apache and Nginx use
-  // /pagespeed_statistics as the handler.
-  statistics_handler_path_.set_default(kNgxPagespeedStatisticsHandlerPath);
 }

 void NgxRewriteOptions::AddProperties() {
@@ -94,6 +120,39 @@ bool NgxRewriteOptions::IsDirective(StringPiece config_directive,
  return StringCaseEqual(config_directive, compare_directive);
 }

+RewriteOptions::OptionScope NgxRewriteOptions::GetOptionScope(
+    StringPiece option_name) {
+  ngx_uint_t i;
+  ngx_uint_t size = sizeof(main_only_options) / sizeof(char*);
+  for (i = 0; i < size; i++) {
+    if (StringCaseEqual(main_only_options[i], option_name)) {
+      return kProcessScopeStrict;
+    }
+  }
+
+  size = sizeof(server_only_options) / sizeof(char*);
+  for (i = 0; i < size; i++) {
+    if (StringCaseEqual(server_only_options[i], option_name)) {
+      return kServerScope;
+    }
+  }
+
+  // This could be made more efficient if RewriteOptions provided a map allowing
+  // access of options by their name. It's not too much of a worry at present
+  // since this is just during initialization.
+  for (OptionBaseVector::const_iterator it = all_options().begin();
+       it != all_options().end(); ++it) {
+    RewriteOptions::OptionBase* option = *it;
+    if (option->option_name() == option_name) {
+      // We treat kProcessScope as kProcessScopeStrict, failing to start if an
+      // option is out of place.
+      return option->scope() == kProcessScope ? kProcessScopeStrict
+                                              : option->scope();
+    }
+  }
+  return kDirectoryScope;
+}
+
 RewriteOptions::OptionSettingResult NgxRewriteOptions::ParseAndSetOptions0(
    StringPiece directive, GoogleString* msg, MessageHandler* handler) {
  if (IsDirective(directive, "on")) {
@@ -144,10 +203,26 @@ RewriteOptions::OptionSettingResult ParseAndSetOptionHelper(
  return RewriteOptions::kOptionOk;
 }

+namespace {
+
+const char* ps_error_string_for_option(
+    ngx_pool_t* pool, StringPiece directive, StringPiece warning) {
+  GoogleString msg =
+      StrCat("\"", directive, "\" ", warning);
+  char* s = string_piece_to_pool_string(pool, msg);
+  if (s == NULL) {
+    return "failed to allocate memory";
+  }
+  return s;
+}
+
+}  // namespace
+
 // Very similar to apache/mod_instaweb::ParseDirective.
 const char* NgxRewriteOptions::ParseAndSetOptions(
    StringPiece* args, int n_args, ngx_pool_t* pool, MessageHandler* handler,
-    NgxRewriteDriverFactory* driver_factory) {
+    NgxRewriteDriverFactory* driver_factory,
+    RewriteOptions::OptionScope scope) {
  CHECK_GE(n_args, 1);

  StringPiece directive = args[0];
@@ -158,6 +233,11 @@ const char* NgxRewriteOptions::ParseAndSetOptions(
    directive.remove_prefix(mod_pagespeed.size());
  }

+  if (GetOptionScope(directive) > scope) {
+    return ps_error_string_for_option(
+        pool, directive, "cannot be set at this scope.");
+  }
+
  GoogleString msg;
  OptionSettingResult result;
  if (n_args == 1) {
@@ -227,25 +307,22 @@ const char* NgxRewriteOptions::ParseAndSetOptions(
    result = ParseAndSetOptionFromName3(
        directive, args[1], args[2], args[3], &msg, handler);
  } else {
-    return "unknown option";
+    return ps_error_string_for_option(
+        pool, directive, "not recognized or too many arguments");
  }

  switch (result) {
    case RewriteOptions::kOptionOk:
      return NGX_CONF_OK;
    case RewriteOptions::kOptionNameUnknown:
-      return "unknown option";
+      return ps_error_string_for_option(
+          pool, directive, "not recognized or too many arguments");
    case RewriteOptions::kOptionValueInvalid: {
-      GoogleString full_directive = "\"";
+      GoogleString full_directive;
      for (int i = 0 ; i < n_args ; i++) {
        StrAppend(&full_directive, i == 0 ? "" : " ", args[i]);
      }
-      StrAppend(&full_directive, "\": ", msg);
-      char* s = string_piece_to_pool_string(pool, full_directive);
-      if (s == NULL) {
-        return "failed to allocate memory";
-      }
-      return s;
+      return ps_error_string_for_option(pool, full_directive, msg);
    }
  }

@@ -40,7 +40,8 @@ class NgxRewriteOptions : public SystemRewriteOptions {
  static void Initialize();
  static void Terminate();

-  NgxRewriteOptions(const StringPiece& description, ThreadSystem* thread_system);
+  NgxRewriteOptions(const StringPiece& description,
+                    ThreadSystem* thread_system);
  explicit NgxRewriteOptions(ThreadSystem* thread_system);
  virtual ~NgxRewriteOptions() { }

@@ -57,7 +58,7 @@ class NgxRewriteOptions : public SystemRewriteOptions {
  // pool is a memory pool for allocating error strings.
  const char* ParseAndSetOptions(
      StringPiece* args, int n_args, ngx_pool_t* pool, MessageHandler* handler,
-      NgxRewriteDriverFactory* driver_factory);
+      NgxRewriteDriverFactory* driver_factory, OptionScope scope);

  // Make an identical copy of these options and return it.
  virtual NgxRewriteOptions* Clone() const;
@@ -116,6 +117,9 @@ class NgxRewriteOptions : public SystemRewriteOptions {
  // ignoring case.
  bool IsDirective(StringPiece config_directive, StringPiece compare_directive);

+  // Returns a given option's scope.
+  RewriteOptions::OptionScope GetOptionScope(StringPiece option_name);
+
  // TODO(jefftk): support fetch proxy in server and location blocks.

  DISALLOW_COPY_AND_ASSIGN(NgxRewriteOptions);
@@ -72,6 +72,7 @@ SystemRequestContext* NgxServerContext::NewRequestContext(

  return new SystemRequestContext(thread_system()->NewMutex(),
                                  timer(),
+                                  ps_determine_host(r),
                                  local_port,
                                  str_to_string_piece(local_ip));
 }
@@ -66,10 +66,10 @@ namespace net_instaweb {
      mutex_(NULL) {
    resolver_timeout_ = resolver_timeout;
    fetch_timeout_ = fetch_timeout;
-    ngx_memzero(&url_, sizeof(url_));
+    ngx_memzero(&proxy_, sizeof(proxy_));
    if (proxy != NULL && *proxy != '\0') {
-      url_.url.data = reinterpret_cast<u_char*>(const_cast<char*>(proxy));
-      url_.url.len = ngx_strlen(proxy);
+      proxy_.url.data = reinterpret_cast<u_char*>(const_cast<char*>(proxy));
+      proxy_.url.len = ngx_strlen(proxy);
    }
    mutex_ = thread_system_->NewMutex();
    log_ = log;
@@ -106,6 +106,36 @@ namespace net_instaweb {
    }
  }

+
+  bool NgxUrlAsyncFetcher::ParseUrl(ngx_url_t* url, ngx_pool_t* pool) {
+    size_t scheme_offset;
+    u_short port;
+    if (ngx_strncasecmp(url->url.data, reinterpret_cast<u_char*>(
+                                     const_cast<char*>("http://")), 7) == 0) {
+      scheme_offset = 7;
+      port = 80;
+    } else if (ngx_strncasecmp(url->url.data, reinterpret_cast<u_char*>(
+                                    const_cast<char*>("https://")), 8) == 0) {
+      scheme_offset = 8;
+      port = 443;
+    } else {
+      scheme_offset = 0;
+      port = 80;
+    }
+
+    url->url.data += scheme_offset;
+    url->url.len -= scheme_offset;
+    url->default_port = port;
+    // See: http://lxr.evanmiller.org/http/source/core/ngx_inet.c#L875
+    url->no_resolve = 0;
+    url->uri_part = 1;
+
+    if (ngx_parse_url(pool, url) == NGX_OK) {
+      return true;
+    }
+    return false;
+  }
+
  // If there are still active requests, cancel them.
  void NgxUrlAsyncFetcher::CancelActiveFetches() {
    // TODO(oschaaf): this seems tricky, this may end up calling
@@ -167,15 +197,15 @@ namespace net_instaweb {
    command_connection_->read->handler = CommandHandler;
    ngx_add_event(command_connection_->read, NGX_READ_EVENT, 0);

-    if (url_.url.len == 0) {
+    if (proxy_.url.len == 0) {
      return true;
    }

    // TODO(oschaaf): shouldn't we do this earlier? Do we need to clean
    // up when parsing the url failed?
-    if (ngx_parse_url(pool_, &url_) != NGX_OK) {
+    if (!ParseUrl(&proxy_, pool_)) {
      ngx_log_error(NGX_LOG_ERR, log_, 0,
-          "NgxUrlAsyncFetcher::Init parse proxy[%V] failed", &url_.url);
+          "NgxUrlAsyncFetcher::Init parse proxy[%V] failed", &proxy_.url);
      return false;
    }
    return true;
@@ -191,7 +221,7 @@ namespace net_instaweb {
  void NgxUrlAsyncFetcher::Fetch(const GoogleString& url,
                                 MessageHandler* message_handler,
                                 AsyncFetch* async_fetch) {
-    async_fetch = EnableInflation(async_fetch, NULL);
+    async_fetch = EnableInflation(async_fetch);
    NgxFetch* fetch = new NgxFetch(url, async_fetch,
          message_handler, fetch_timeout_, log_);
    ScopedMutex lock(mutex_);
@@ -115,13 +115,14 @@ class NgxUrlAsyncFetcher : public UrlAsyncFetcher {

 private:
  static void TimeoutHandler(ngx_event_t* tev);
+  static bool ParseUrl(ngx_url_t* url, ngx_pool_t* pool);
  friend class NgxFetch;

  NgxFetchPool active_fetches_;
  // Add the pending task to this list
  NgxFetchPool pending_fetches_;
  NgxFetchPool completed_fetches_;
-  ngx_url_t url_;
+  ngx_url_t proxy_;

  int fetchers_count_;
  bool shutdown_;
@@ -5,7 +5,7 @@
 worker_processes  1;

 daemon @@DAEMON@@;
-master_process @@MASTER_PROCESS@@;
+master_process on;

 error_log "@@ERROR_LOG@@" debug;
 pid "@@TEST_TMP@@/nginx.pid";
@@ -28,10 +28,22 @@ http {
  proxy_temp_path "@@TMP_PROXY_CACHE@@";

  root "@@SERVER_ROOT@@";
+
+  # Block 5a: Decide on Cache-Control header value to use for outgoing
+  # response.
+  # Map new_cache_control_header_val to "no-cache, max-age=0" if the
+  # content is html and use the original Cache-Control header value
+  # in all other cases.
+  map $upstream_http_content_type $new_cache_control_header_val {
+    default $upstream_http_cache_control;
+    "~*text/html" "no-cache, max-age=0";
+  }
+
  pagespeed UsePerVHostStatistics on;
  pagespeed InPlaceResourceOptimization on;
  pagespeed CreateSharedMemoryMetadataCache "@@SHM_CACHE@@" 8192;
  pagespeed PreserveUrlRelativity on;
+  pagespeed BlockingRewriteKey psatest;

  # CriticalImagesBeaconEnabled is now on by default, but we disable in testing.
  # With this option enabled, the inline image system test will currently fail.
@@ -40,6 +52,11 @@ http {
  # critical images to be inlined, so we just disable the option here.
  pagespeed CriticalImagesBeaconEnabled false;

+  # By default, resources will not be used for inlining without explicit
+  # authorization. Supported values are off or a comma-separated list of strings
+  # from {Script,Stylesheet}.
+  pagespeed InlineResourcesWithoutExplicitAuthorization off;
+
  pagespeed Statistics on;
  pagespeed StatisticsLogging on;
  pagespeed LogDir "@@TEST_TMP@@/logdir";
@@ -53,7 +70,6 @@ http {
    server_name max-cacheable-content-length.example.com;
    pagespeed FileCachePath "@@FILE_CACHE@@";

-    pagespeed BlockingRewriteKey psatest;

    pagespeed RewriteLevel PassThrough;
    pagespeed EnableFilters rewrite_javascript;
@@ -64,6 +80,7 @@ http {
  @@RESOLVER@@

  server {
+    # Block 1:  Basic port, server_name definitions.
    # This server represents the external caching layer server which
    # receives user requests and proxies them to the upstream server
    # running on the PRIMARY_PORT when the response is not available in
@@ -72,58 +89,149 @@ http {
    server_name proxy_cache.example.com;
    pagespeed FileCachePath "@@FILE_CACHE@@";

+    # Disable PageSpeed on this server.
    pagespeed off;

-    set $ua_dependent_ps_capability_list "";
-    set $bypass_cache 1;
+    # Block 2: Define prefix for proxy_cache_key based on the UserAgent.
+
+    # Define placeholder PS-CapabilityList header values for large and small
+    # screens with no UA dependent optimizations. Note that these placeholder
+    # values should not contain any of ll, ii, dj, jw or ws, since these
+    # codes will end up representing optimizations to be supported for the
+    # request.
+    set $default_ps_capability_list_for_large_screens "LargeScreen.SkipUADependentOptimizations";
+    set $default_ps_capability_list_for_small_screens "TinyScreen.SkipUADependentOptimizations";
+
+    # As a fallback, the PS-CapabilityList header that is sent to the upstream
+    # PageSpeed server should be for a large screen device with no browser
+    # specific optimizations.
+    set $ps_capability_list $default_ps_capability_list_for_large_screens;
+
+    # Cache-fragment 1: Desktop User-Agents that support lazyload_images (ll),
+    # inline_images (ii) and defer_javascript (dj).
+    # Note: Wget is added for testing purposes only.
    if ($http_user_agent ~* "Chrome/|Firefox/|MSIE |Safari|Wget") {
-      # User Agents that support lazyload-images (ll), inline-images (ii) and
-      # defer-javascript (dj).
-      set $ua_dependent_ps_capability_list "ll,ii,dj:";
-      set $bypass_cache 0;
+      set $ps_capability_list "ll,ii,dj:";
    }
+    # Cache-fragment 2: Desktop User-Agents that support lazyload_images (ll),
+    # inline_images (ii), defer_javascript (dj), webp (jw) and lossless_webp
+    # (ws).
    if ($http_user_agent ~*
        "Chrome/[2][3-9]+\.|Chrome/[[3-9][0-9]+\.|Chrome/[0-9]{3,}\.") {
-      # User Agents that support lazyload-images (ll), inline-images (ii),
-      # defer-javascript (dj), webp (jw) and webp-lossless (ws).
-      set $ua_dependent_ps_capability_list "ll,ii,dj,jw,ws:";
-      set $bypass_cache 0;
+      set $ps_capability_list "ll,ii,dj,jw,ws:";
+    }
+    # Cache-fragment 3: This fragment contains (a) Desktop User-Agents that
+    # match fragments 1 or 2 but should not because they represent older
+    # versions of certain browsers or bots and (b) Tablet User-Agents that
+    # correspond to large screens. These will only get optimizations that work
+    # on all browsers and use image compression qualities applicable to large
+    # screens. Note that even Tablets that are capable of supporting inline or
+    # webp images, e.g. Android 4.1.2, will not get these advanced
+    # optimizations.
+    if ($http_user_agent ~* "Firefox/[1-2]\.|MSIE [5-8]\.|bot|Yahoo!|Ruby|RPT-HTTPClient|(Google \(\+https\:\/\/developers\.google\.com\/\+\/web\/snippet\/\))|Android|iPad|TouchPad|Silk-Accelerated|Kindle Fire") {
+      set $ps_capability_list $default_ps_capability_list_for_large_screens;
+    }
+    # Cache-fragment 4: Mobiles and small screen Tablets will use image compression
+    # qualities applicable to small screens, but all other optimizations will be
+    # those that work on all browsers.
+    if ($http_user_agent ~* "Mozilla.*Android.*Mobile*|iPhone|BlackBerry|Opera Mobi|Opera Mini|SymbianOS|UP.Browser|J-PHONE|Profile/MIDP|portalmmm|DoCoMo|Obigo|Galaxy Nexus|GT-I9300|GT-N7100|HTC One|Nexus [4|7|S]|Xoom|XT907") {
+      set $ps_capability_list $default_ps_capability_list_for_small_screens;
    }

-    # All User Agents that represent
-    # 1) mobiles
-    # 2) tablets
-    # 3) desktop browsers that do not have defer-javascript capability at a minimum
-    # are made to go to the pagespeed server directly bypassing the proxy_cache.
-    if ($http_user_agent ~* "Firefox/[1-2]\.|MSIE [5-8]\.") {
-      set $ua_dependent_ps_capability_list "";
-      set $bypass_cache 1;
+    # Block 3a: Bypass the cache for .pagespeed. resource. PageSpeed has its own
+    # cache for these, and these could bloat up the caching layer.
+    if ($uri ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+") {
+      set $bypass_cache "1";
    }
-    if ($http_user_agent ~* "Mozilla.*Android.*Mobile*|iPhone|BlackBerry|Opera Mobi|Opera Mini|SymbianOS|UP.Browser|J-PHONE|Profile/MIDP|portalmmm|DoCoMo|Obigo") {
-      # These are Mobile User Agents. We don't cache responses for these.
-      set $ua_dependent_ps_capability_list "";
-      set $bypass_cache 1;
-    }
-    if ($http_user_agent ~* "Android|iPad|TouchPad|Silk-Accelerated|Kindle Fire") {
-      # These are Tablet User Agents. We don't cache responses for these.
-      set $ua_dependent_ps_capability_list "";
+
+    set_random $rand 0 100;
+    set $should_beacon_header_val "";
+    if ($rand ~* "^[0-4]$") {
+      set $should_beacon_header_val "random_rebeaconing_key";
      set $bypass_cache 1;
    }

+    # Block 3b: Only cache responses to clients that support gzip.  Most clients
+    # do, and the cache holds much more if it stores gzipped responses.
+    if ($http_accept_encoding !~* gzip) {
+      set $bypass_cache "1";
+    }
+
+    # Block 4: Location block for purge requests.
    location ~ /purge(/.*) {
-      allow all;
-      proxy_cache_purge htmlcache $ua_dependent_ps_capability_list$1$is_args$args;
+      allow 127.0.0.1;
+      deny all;
+      proxy_cache_purge htmlcache $ps_capability_list$1$is_args$args;
    }

+    # Block 6: Location block with proxy_cache directives.
    location /mod_pagespeed_test/cachable_rewritten_html/ {
+      # 1: Upstream PageSpeed server is running at localhost:8050.
      proxy_pass http://localhost:@@PRIMARY_PORT@@;
-      proxy_set_header Host $host;
-      proxy_cache_valid 200 30s;
+      # 2: Use htmlcache as the zone for caching.
      proxy_cache htmlcache;
-      proxy_ignore_headers Cache-Control;
-      add_header X-Cache $upstream_cache_status;
-      proxy_cache_key $ua_dependent_ps_capability_list$uri$is_args$args;
+      # 3: Bypass requests that correspond to .pagespeed. resources
+      # or clients that do not support gzip etc.
      proxy_cache_bypass $bypass_cache;
+      # 4: Use the redefined proxy_cache_key and make sure the /purge/
+      # block uses the same key.
+      proxy_cache_key $ps_capability_list$uri$is_args$args;
+      # 5: Forward Host header to upstream server.
+      proxy_set_header Host $host;
+      # 6: Set the PS-CapabilityList header for PageSpeed server to respect.
+      proxy_set_header PS-CapabilityList $ps_capability_list;
+      add_header PS-CapabilityList $ps_capability_list;
+      # 7: Add a header for identifying cache hits/misses/expires. This is
+      # for debugging purposes only and can be commented out in production.
+      add_header X-Cache $upstream_cache_status;
+
+      # Block 5b: Override Cache-Control headers as needed.
+      # Hide the upstream cache control header.
+      proxy_hide_header Cache-Control;
+      # Add the inferred Cache-Control header.
+      add_header Cache-Control $new_cache_control_header_val;
+
+      proxy_set_header PS-ShouldBeacon $should_beacon_header_val;
+      proxy_hide_header PS-ShouldBeacon;
+    }
+  }
+
+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name if-in-server.example.com;
+    pagespeed FileCachePath "@@SECONDARY_CACHE@@";
+
+    pagespeed RewriteLevel PassThrough;
+    set $inline_javascript "No";
+
+    if ($http_x_custom_header_inline_js) {
+      # TODO(jefftk): Turn on NGX_HTTP_SIF_CONF and figure out how to get
+      # pagespeed directives inside of a server location block to be respected,
+      # then uncomment the following line and duplicate the if-in-location test
+      # for if-in-server.
+      #pagespeed EnableFilters inline_javascript;
+      set $inline_javascript "Yes";
+    }
+
+    add_header "X-Inline-Javascript" $inline_javascript;
+  }
+
+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name if-in-location.example.com;
+    pagespeed FileCachePath "@@SECONDARY_CACHE@@";
+
+
+    location / {
+      set $inline_javascript "No";
+      pagespeed RewriteLevel PassThrough;
+
+      if ($http_x_custom_header_inline_js) {
+        pagespeed EnableFilters inline_javascript;
+        set $inline_javascript "Yes";
+      }
+
+      add_header "X-Inline-Javascript" $inline_javascript;
    }
  }

@@ -141,6 +249,7 @@ http {
    listen @@SECONDARY_PORT@@;
    server_name experiment.example.com;
    pagespeed FileCachePath "@@FILE_CACHE@@";
+    pagespeed InPlaceResourceOptimization off;

    pagespeed RunExperiment on;
    pagespeed AnalyticsID "123-45-6734";
@@ -148,6 +257,7 @@ http {
    pagespeed ExperimentSpec
      "id=7;enable=recompress_images;disable=convert_jpeg_to_progressive;percent=50";
    pagespeed ExperimentSpec "id=2;enable=recompress_images;percent=50";
+    pagespeed ExperimentSpec "id=3;default;percent=0";
  }
  server {
    listen @@SECONDARY_PORT@@;
@@ -159,6 +269,7 @@ http {
    pagespeed ExperimentSpec
      "id=7;enable=recompress_images;disable=convert_jpeg_to_progressive;percent=50";
    pagespeed ExperimentSpec "id=2;enable=recompress_images;percent=50";
+    pagespeed ExperimentSpec "id=3;default;percent=0";
  }

  server {
@@ -200,6 +311,23 @@ http {
    pagespeed CriticalImagesBeaconEnabled true;
  }

+  server {
+    # Setup a vhost with the critical image beacon enabled to make sure that
+    # downstream caches and rebeaconing interact correctly.
+    listen @@SECONDARY_PORT@@;
+    server_name downstreamcacherebeacon.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@";
+
+    pagespeed RewriteLevel PassThrough;
+    pagespeed CriticalImagesBeaconEnabled true;
+    # Enable the downstream caching feature and specify a rebeaconing key.
+    pagespeed DownstreamCachePurgeLocationPrefix "http://localhost:@@SECONDARY_PORT@@/purge";
+    pagespeed DownstreamCacheRebeaconingKey random_rebeaconing_key;
+    location ~ .*[.]html {
+      add_header Cache-Control "private, max-age=3000";
+    }
+  }
+
  server {
    listen @@SECONDARY_PORT@@;
    server_name renderedimagebeacon.example.com;
@@ -210,6 +338,60 @@ http {
    pagespeed CriticalImagesBeaconEnabled true;
  }

+
+  # Build a configuration hierarchy where at the root we have turned on
+  # OptimizeForBandwidth, and in various subdirectories we override settings
+  # to make them more aggressive.
+  #
+  # In Apache we can do this all with Directory blocks, but to get the same
+  # inheretence in Nginx we need to have location blocks inside a server block.
+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name optimizeforbandwidth.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@";
+
+    pagespeed RewriteLevel OptimizeForBandwidth;
+    pagespeed DisableFilters add_instrumentation;
+
+    location /mod_pagespeed_test/optimize_for_bandwidth/inline_css {
+      pagespeed EnableFilters inline_css;
+    }
+    location /mod_pagespeed_test/optimize_for_bandwidth/css_urls {
+      pagespeed CssPreserveURLs off;
+    }
+    location /mod_pagespeed_test/optimize_for_bandwidth/image_urls {
+      pagespeed ImagePreserveURLs off;
+    }
+    location /mod_pagespeed_test/optimize_for_bandwidth/core_filters {
+      pagespeed RewriteLevel CoreFilters;
+    }
+  }
+
+  server {
+    # For testing with a custom origin header.  In this VirtualHost,
+    # /mod_pagespeed_test is included in our DocumentRoot and thus does
+    # not need to be in any resource URL paths.  This helps us verify that
+    # we are looping back to the corect VirtualHost -- if we hit the wrong
+    # one it will not work.  Also we don't have a VirtualHost for
+    # sharedcdn.example.com, so the default Host header used for
+    # origin-mapping won't work either.  Instead, we want origin-fetches
+    # to go back to this VirtualHost so we rely on the new third optional
+    # argument to MapOriginDomain.
+    listen @@SECONDARY_PORT@@;
+    server_name customhostheader.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@_test";
+    root "@@SERVER_ROOT@@/mod_pagespeed_test";
+
+    pagespeed on;
+    pagespeed RewriteLevel PassThrough;
+    pagespeed EnableFilters rewrite_images;
+    # Don't use localhost, as ngx_pagespeed's native fetcher cannot resolve it
+    pagespeed MapOriginDomain 127.0.0.1:@@SECONDARY_PORT@@/customhostheader
+              sharedcdn.example.com/test customhostheader.example.com;
+    pagespeed JpegRecompressionQuality 50;
+    pagespeed CriticalImagesBeaconEnabled false;
+  }
+
  server {
    # Sets up a virtual host where we can specify forbidden filters without
    # affecting any other hosts.
@@ -217,7 +399,6 @@ http {
    server_name forbidden.example.com;
    pagespeed FileCachePath "@@FILE_CACHE@@";

-    pagespeed BlockingRewriteKey psatest;

    # Start with all core filters enabled ...
    pagespeed RewriteLevel CoreFilters;
@@ -228,6 +409,16 @@ http {
    pagespeed DisableFilters inline_css;
  }

+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name unauthorizedresources.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@";
+
+    pagespeed RewriteLevel PassThrough;
+    pagespeed InlineResourcesWithoutExplicitAuthorization Script,Stylesheet;
+    pagespeed CssInlineMaxBytes 1000000;
+  }
+
  server {
    listen @@SECONDARY_PORT@@;
    server_name client-domain-rewrite.example.com;
@@ -342,7 +533,7 @@ http {
  # Note that we test with two distinct caches.
  server {
    listen @@SECONDARY_PORT@@;
-    server_name embed-config-html.example.com;
+    server_name embed-config-html.example.org;
    pagespeed FileCachePath "@@FILE_CACHE@@";

    root "@@SERVER_ROOT@@/mod_pagespeed_test";
@@ -352,6 +543,9 @@ http {
    pagespeed DisableFilters inline_css,extend_cache,inline_javascript;
    pagespeed Domain embed-config-resources.example.com;

+    # Share a cache keyspace with embed-config-resources.example.com.
+    pagespeed CacheFragment "embed-config";
+
    pagespeed LoadFromFile "http://embed-config-resources.example.com/"
      "@@SERVER_ROOT@@/mod_pagespeed_example/";
  }
@@ -367,12 +561,26 @@ http {

    # Note that we do not set the jpeg quality here, but take
    # it from image URL query parameters that we synthesize in
-    # from embed-config-html.example.com.
+    # from embed-config-html.example.org.
+
+    # Share a cache keyspace with embed-config-html.example.org.
+    pagespeed CacheFragment "embed-config";

    pagespeed LoadFromFile "http://embed-config-resources.example.com/"
      "@@SERVER_ROOT@@/mod_pagespeed_example/";
  }

+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name ipro-for-browser.example.com;
+    root "@@SERVER_ROOT@@/mod_pagespeed_example";
+    pagespeed EnableFilters rewrite_images,rewrite_css;
+    pagespeed EnableFilters convert_to_webp_lossless;
+    pagespeed EnableFilters in_place_optimize_for_browser;
+    pagespeed InPlaceResourceOptimization on;
+    pagespeed FileCachePath "@@IPRO_CACHE@@";
+  }
+
  server {
    listen @@SECONDARY_PORT@@;
    server_name respectvary.example.com;
@@ -424,6 +632,7 @@ http {
    pagespeed MapRewriteDomain cdn.example.com origin.example.com;
    pagespeed RewriteLevel PassThrough;
    pagespeed EnableFilters rewrite_css,rewrite_images;
+    pagespeed CacheFragment "example";
  }
  server {
    # Sets up a logical origin for CDNs to fetch content from, on
@@ -436,6 +645,7 @@ http {
    pagespeed MapRewriteDomain cdn.example.com origin.example.com;
    pagespeed RewriteLevel PassThrough;
    pagespeed EnableFilters rewrite_css,rewrite_images;
+    pagespeed CacheFragment "example";
  }
  server {
    # Sets up a logical cdn, which is where we tell browsers to fetch resources
@@ -491,7 +701,6 @@ http {
    server_name blocking.example.com;
    pagespeed FileCachePath "@@SECONDARY_CACHE@@";

-    pagespeed BlockingRewriteKey psatest;
    pagespeed RewriteLevel PassThrough;
    pagespeed EnableFilters rewrite_images;
  }
@@ -561,6 +770,11 @@ http {
    location /mod_pagespeed_test/ipro/nocache/test_image_dont_reuse.png {
      add_header Cache-Control no-cache;
    }
+
+    location /mod_pagespeed_test/ipro/no-cache-control-header {
+      add_header Cache-Control "";
+      pagespeed ImplicitCacheTtlMs 333000;
+    }
  }

  # Test hosts to cover all possible cache configurations.  L1 will be filecache
@@ -615,6 +829,36 @@ http {
    pagespeed CriticalImagesBeaconEnabled false;
  }

+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name date.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@";
+    add_header "Date" "Date: Fri, 16 Oct 2009 23:05:07 GMT";
+  }
+
+  # Proxy + IPRO a gzip'd file for testing Issue 896.
+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name ipro-proxy.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@_ipro_proxy";
+
+    pagespeed on;
+    pagespeed InPlaceResourceOptimization on;
+    pagespeed EnableFilters rewrite_domains;
+
+    location / {
+      proxy_pass
+       "http://localhost:@@PRIMARY_PORT@@/mod_pagespeed_test/ipro/mod_deflate/";
+    }
+  }
+
+  server {
+    listen @@SECONDARY_PORT@@;
+    server_name compressed-css.example.com;
+    pagespeed FileCachePath "@@FILE_CACHE@@";
+    pagespeed InPlaceResourceOptimization on;
+  }
+
  server {
    listen       @@PRIMARY_PORT@@;
    server_name  localhost;
@@ -624,16 +868,24 @@ http {
      add_header "" "";
    }

+    # Backend for ipro-proxy.example.com
+    location /mod_pagespeed_test/ipro/mod_deflate/ {
+      pagespeed off;
+    }
+
    location /mod_pagespeed_test/cachable_rewritten_html/ {
      # This location has the html files that will be configured to be stored
      # in the proxy_cache layer.
      pagespeed DownstreamCachePurgeMethod "GET";
      pagespeed DownstreamCachePurgeLocationPrefix "http://localhost:@@SECONDARY_PORT@@/purge";
+      pagespeed DownstreamCacheRebeaconingKey "random_rebeaconing_key";
      # We use a very small deadline here to force the rewriting to not complete
      # in the very first attempt.
      pagespeed RewriteDeadlinePerFlushMs 1;
      pagespeed RewriteLevel PassThrough;
-      pagespeed EnableFilters collapse_whitespace,extend_cache,recompress_images,convert_jpeg_to_webp,defer_javascript;
+      pagespeed EnableFilters collapse_whitespace,extend_cache,recompress_images;
+      pagespeed CriticalImagesBeaconEnabled true;
+      add_header Cache-Control "public, max-age=100";
    }

    location /mod_pagespeed_test/disable_no_transform/index.html {
@@ -649,6 +901,7 @@ http {
    #pagespeed MemcachedThreads 1;

    pagespeed on;
+    pagespeed MessageBufferSize 200000;

    #pagespeed CacheFlushPollIntervalSec 1;

@@ -660,14 +913,6 @@ http {
    pagespeed Library 43 1o978_K0_LNE5_ystNklf
    http://www.modpagespeed.com/rewrite_javascript.js;

-    # If X-PSA-Blocking-Rewrite request header is present and its value matches
-    # the value of BlockingRewriteKey below, the response will be fully
-    # rewritten before being flushed to the client.
-    pagespeed BlockingRewriteKey psatest;
-
-    # Disable parsing if the size of the HTML exceeds 50kB.
-    pagespeed MaxHtmlParseBytes 50000;
-
    add_header X-Extra-Header 1;

    # Establish a proxy mapping where the current server proxies an image
@@ -706,6 +951,10 @@ http {
      pagespeed DisableFilters collapse_whitespace;
    }

+    location /mod_pagespeed_test/max_html_parse_size {
+      pagespeed MaxHtmlParseBytes 5000;
+    }
+
    location ~ \.php$ {
      fastcgi_param SCRIPT_FILENAME $request_filename;
      fastcgi_param QUERY_STRING $query_string;
@@ -781,6 +1030,71 @@ http {
      expires 5m;
    }

+    location /mod_pagespeed_test/ipro/instant/wait/ {
+      pagespeed InPlaceWaitForOptimized on;
+
+      # TODO(jefftk): When running with valgrind we need to raise the rewrite
+      # deadline or else we get "Deadline exceeded for rewrite of resource".
+      # I had thought InPlaceWaitForOptimized was supposed to disable the
+      # rewrite deadline, but it seems not to.
+      #
+      # InPlaceWaitForOptimized is definitely doing something, though: when I
+      # remove the InPlaceRewriteDeadlineMs line below and run without valgrind,
+      # purple.css is optimized via ipro on the first request but if I set
+      # InPlaceWaitForOptimized to off then it is not.
+      pagespeed InPlaceRewriteDeadlineMs 1000;
+    }
+
+    location /mod_pagespeed_test/ipro/instant/deadline/ {
+      pagespeed InPlaceRewriteDeadlineMs -1;
+    }
+
+    # Test to make sure that user-authenticated resources do not get cached and
+    # optimized.
+    location /mod_pagespeed_test/auth/ {
+       auth_basic "Restricted";
+       auth_basic_user_file "@@SERVER_ROOT@@mod_pagespeed_test/auth/passwd.conf";
+    }
+
+    location /mod_pagespeed_test/ipro/cookie/ {
+      # Add Vary:Cookie.  This should prevent us from optimizing the
+      # vary_cookie.css even though PagespeedRespectVary is off.
+      # test/nginx_system_test.sh does the fetches test with and w/o cookies.
+      add_header Vary Cookie;
+      pagespeed RespectVary off;
+      pagespeed InPlaceWaitForOptimized on;
+    }
+
+    location /mod_pagespeed_test/ipro/cookie2/ {
+      # Add Vary:Cookie2.  This should prevent us from optimizing the
+      # vary_cookie2.css even though PagespeedRespectVary is off.
+      # test/nginx_system_test.sh does the fetches test with and w/o cookie2.
+      add_header Vary Cookie2;
+      pagespeed RespectVary off;
+      pagespeed InPlaceWaitForOptimized on;
+    }
+
+    location /mod_pagespeed_test/vary/ {
+      pagespeed RespectVary on;
+    }
+
+    location ~ /mod_pagespeed_test/vary/\.(js|css)$ {
+        add_header Vary User-Agent;
+    }
+
+    location /mod_pagespeed_test/vary/no_respect/ {
+      pagespeed DisableFilters add_instrumentation,inline_css;
+      pagespeed RespectVary off;
+    }
+
+    location /mod_pagespeed_test/experimental_js_minifier/ {
+      pagespeed UseExperimentalJsMinifier on;
+    }
+
+    pagespeed LoadFromFile
+        "http://localhost:@@PRIMARY_PORT@@/mod_pagespeed_test/ipro/instant/"
+        "@@SERVER_ROOT@@/mod_pagespeed_test/ipro/instant/";
+
    pagespeed EnableFilters remove_comments;

    # Test LoadFromFile mapping by mapping one dir to another.
@@ -819,6 +1133,7 @@ http {
  # set up gzip
  gzip  on;
  gzip_vary on;
+  gzip_proxied any;
  # Turn on gzip for all content types that should benefit from it.
  gzip_types application/ecmascript;
  gzip_types application/javascript;
@@ -0,0 +1,165 @@
+# Copyright 2013 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Author: oschaaf@we-amp.com (Otto van der Schaaf)
+
+
+# The first few suppressions can be found in other modules
+# and easily found when searched for, and seem false positives.
+{
+   <nginx false positive>
+   Memcheck:Param
+   socketcall.sendmsg(msg.msg_iov[i])
+   fun:__sendmsg_nocancel
+   fun:ngx_write_channel
+   fun:ngx_signal_worker_processes
+   fun:ngx_master_process_cycle
+   fun:main
+}
+{
+   <nginx false positive>
+   Memcheck:Param
+   socketcall.sendmsg(msg.msg_iov[i])
+   fun:__sendmsg_nocancel
+   fun:ngx_write_channel
+   fun:ngx_master_process_cycle
+   fun:main
+}
+{
+   <nginx false positive>
+   Memcheck:Param
+   socketcall.sendmsg(msg.msg_iov[i])
+   fun:__sendmsg_nocancel
+   fun:ngx_write_channel
+   fun:ngx_pass_open_channel
+   fun:ngx_start_cache_manager_processes
+   fun:ngx_master_process_cycle
+   fun:main
+}
+{
+   <nginx false positive>
+   Memcheck:Param
+   socketcall.sendmsg(msg.msg_iov[i])
+   fun:__sendmsg_nocancel
+   fun:ngx_write_channel
+   fun:ngx_pass_open_channel
+   fun:ngx_start_cache_manager_processes
+   fun:ngx_master_process_cycle
+   fun:main
+}
+{
+   <nginx false positive>
+   Memcheck:Leak
+   fun:malloc
+   fun:ngx_alloc
+   fun:ngx_event_process_init
+   fun:ngx_worker_process_init
+   fun:ngx_worker_process_cycle
+   fun:ngx_spawn_process
+   fun:ngx_start_worker_processes
+   fun:ngx_master_process_cycle
+   fun:main
+}
+{
+   <nginx false positive>
+   Memcheck:Param
+   socketcall.sendmsg(msg.msg_iov[i])
+   fun:__sendmsg_nocancel
+   fun:ngx_write_channel
+   fun:ngx_pass_open_channel
+   fun:ngx_start_worker_processes
+   fun:ngx_master_process_cycle
+   fun:main
+}
+
+# similar to http://trac.nginx.org/nginx/ticket/369
+{
+   <nginx false positive>
+   Memcheck:Param
+   pwrite64(buf)
+   obj:/lib/x86_64-linux-gnu/libpthread-2.15.so
+   fun:ngx_write_file
+   fun:ngx_write_chain_to_file
+   fun:ngx_write_chain_to_temp_file
+   fun:ngx_event_pipe_write_chain_to_temp_file
+   fun:ngx_event_pipe
+   fun:ngx_http_upstream_process_upstream
+   fun:ngx_http_upstream_process_header
+   fun:ngx_http_upstream_handler
+   fun:ngx_epoll_process_events
+   fun:ngx_process_events_and_timers
+   fun:ngx_worker_process_cycle
+}
+# Mentioned in https://github.com/pagespeed/ngx_pagespeed/issues/103
+# Assuming a false postives as the issue is closed.
+{
+   <nginx false positive>
+   Memcheck:Param
+   write(buf)
+   obj:/lib/x86_64-linux-gnu/libpthread-2.15.so
+   fun:ngx_log_error_core
+   fun:ngx_http_parse_complex_uri
+   fun:ngx_http_process_request_uri
+   fun:ngx_http_process_request_line
+   fun:ngx_http_wait_request_handler
+   fun:ngx_epoll_process_events
+   fun:ngx_process_events_and_timers
+   fun:ngx_worker_process_cycle
+   fun:ngx_spawn_process
+   fun:ngx_start_worker_processes
+   fun:ngx_master_process_cycle
+}
+
+# Extra suppresions for testing in release mode:
+
+{
+   <re2 uninitialised value in optimized code>
+   Memcheck:Cond
+   fun:_ZN3re24Prog8OptimizeEv
+   ...
+}
+{
+   <re2 uninitialised value in optimized code>
+   Memcheck:Value8
+   fun:_ZN3re24Prog8OptimizeEv
+   ...
+}
+
+{
+   <re2 uninitialised value in optimized code>
+   Memcheck:Cond
+   fun:_ZN3re2L4AddQEPNS_9SparseSetEi
+   ...
+}
+
+{
+   <re2 uninitialised value in optimized code>
+   Memcheck:Value8
+   fun:_ZN3re2L4AddQEPNS_9SparseSetEi
+   ...
+}
+
+{
+   <re2 uninitialized value in optimized code>
+   Memcheck:Value8
+   fun:_ZN3re23DFA10AddToQueueEPNS0_5WorkqEij
+   ...
+}
+
+{
+   <re2 uninitialized value in optimized code>
+   Memcheck:Cond
+   fun:_ZN3re23DFA10AddToQueueEPNS0_5WorkqEij
+   ...
+}
Author	SHA1	Message	Date
Maks Orlovich	c3bff67637	Fix improper cache key on write for IPRO of resources with PageSpeed params We need to be writing with the base name, with params stripped. This was causing a test to "flake" under valgrind, since with it we got to the test showing the bug slowly enough for the entry written out by an earlier test under proper URL to expire.	2014-04-02 18:41:08 +02:00
Maks Orlovich	5a1a29ee8b	trunk-tracking: update from r3872 to r3895 Squash-merge of @morlovich's work on this. Note that this does not yet incorprorate the new pagespeed_admin handler from r3891	2014-04-02 18:39:08 +02:00
Jeff Kaufman	61025c5e4f	if: support pagespeed directives in location if blocks	2014-03-31 10:35:05 -04:00
Otto van der Schaaf	d3e7900704	Merge pull request #648 from pagespeed/jmaessen-trunk-tracking trunk-tracking: update from r3852 to r3872 note that it will only compile against mod_pagespeed r3872 if you first patch in r3889	2014-03-28 22:37:31 +01:00
Jan-Willem Maessen	85e19c962b	trunk-tracking: update from r3852 to r3872	2014-03-27 10:59:13 -04:00
Otto van der Schaaf	20fd7d18dd	Merge pull request #646 from pagespeed/oschaaf-chunked-transfer content-length: fix pagespeed/ipro resource content-lengths	2014-03-26 08:53:50 +01:00
Otto van der Schaaf	4667aa1fc3	content-length: fix pagespeed/ipro resource content-lengths Makes life downstream easier and more efficient	2014-03-25 21:29:28 +01:00
Jeff Kaufman	bfaf4268cf	Merge pull request #639 from pagespeed/jefftk-trunk-tracking trunk-tracking: update from r3827 to r3852	2014-03-19 16:57:28 -04:00
Jeff Kaufman	215071a383	trunk-tracking: update from r3827 to r3852 * port EnrollExperiment tests. * tricky changes to get gzip+ipro working properly. * reduce code duplication by calling copy_response_headers_from_ngx instead of reimplementing it. * port Vary: Accept test changes.	2014-03-18 16:58:50 -04:00
Jeff Kaufman	630f7d848b	lint: fix lint errors	2014-03-18 10:38:58 -04:00
Huibao Lin	d90245af69	trunk-tracking: merge changes in from master: * release: prepare 1.7.30.3 * Don't call chown() when initializing config dirs unless owner != worker user. * Update README.md * release: build against backported change * release: udpate version number 1.7.30.3 to 1.7.30.4	2014-03-18 10:38:28 -04:00
Jeff Kaufman	c754cf1be0	trunk-tracking: update from r3811 to r3927	2014-03-07 11:18:35 -05:00
Jeff Kaufman	9e4d26da5e	Merge pull request #629 from pagespeed/jmaessen-trunk-tracking trunk-tracking: update from r3784 to r3811	2014-03-03 16:56:15 -05:00
Jan-Willem Maessen	027d481817	trunk-tracking: update from r3784 to r3811 The big change here is Vary: headers, which are no longer being stripped from the pagespeed response. Take a look; it might be we want to be smart about merging these headers with the original request's Vary: headers.	2014-02-27 19:41:23 -05:00
Jeff Kaufman	67106e920e	apr: call SystemRewriteDriverFactory::InitApr() squash-merge of morlovich's #601.	2014-02-20 09:13:56 -05:00
Jeff Kaufman	207eb2d131	Merge pull request #622 from pagespeed/jefftk-trunk-tracking trunk-tracking: update from r3770 to r3784	2014-02-19 12:58:40 -05:00
Jeff Kaufman	d9e2142f31	trunk-tracking: update from r3770 to r3784 * r3772 changed the format of unauthorized resource inlining config * r3776 changed query param handling, but due to Jud's nice work in #616 needed no changes. * r3780 huibao's refactoring lets us remove some duplicate code * r3783 changes header passing with ipro * Also cleaned up error messages for unknown options.	2014-02-19 10:53:48 -05:00
Jud Porter	33d006625e	Merge pull request #616 from pagespeed/jud-trunk-tracking trunk-tracking: update from r3736 to r3770	2014-02-11 16:43:33 -05:00
Anupama Dutta	2dcaa95278	trunk-tracking: update from r3736 to r3770 Updated global_only options to include the correct APACHE_CONFIG_OPTIONX directives. Removed repeated tests for prioritize_critical_css basic functionality. Added new tests, mostly downstream caching tests and related pagespeed.conf updates. Also added missing pagespeed.conf updates for downstream caching. Added support for checking option scope.	2014-02-11 16:40:07 -05:00
Jeff Kaufman	f5252b569a	trunk-tracking: update from r3715 to r3736 Squash-merge of Jan's #608 and Otto's #611. * r3726: * Updated closure compiler flags for static JS files. * r3729: * Centralize parsing of FetchHttps in SystemRewriteOptions so ngx_pagespeed can get it too. * To keep the helpful error_message from SerfUrlAsyncFetcher, wire it through RewriteOptions as a new-fangled error_detail. * r3735: * Follow-up changes for downstream caching integration with beaconing dependent filters: If a downstream cache rebeaconing key is configured, we should instrument the page only if the key present in the PS-ShouldBeacon header matches the one in the configuration. This allows us to send no-cache headers for anything that carries the right beaconing key, and continue to send out the original cache control headers in other cases where downstream caching is enabled. * Native fetcher: fortify handling of content length (and absense). * Native fetcher: fail when the stream terminates before having completely parsed the headers. * Tests: Rename `test_filter` -> `start_test` in ngx_system_test.sh for a test. * Tests: Move blockingrewrite key to the http {} block. * Tests: Update localhost -> 127.0.0.1. The native fetcher uses dns to resolve, and won't be able to retreive an ip for localhost. * Tests: Allow outstanding proxy fetches some time to finish when running under valgrind, before terminating nginx. * Valgrind: Add suppressions to make testing a release build pass. This pull update was extra work because the valgrind and native fetcher flows had rotted a bit. We need to make sure to test them with every update.	2014-01-31 10:07:36 -05:00
Jeff Kaufman	4783144e7d	Merge branch 'master' into trunk-tracking	2014-01-24 16:56:26 -05:00
Jeff Kaufman	83205c9c31	Merge pull request #606 from pagespeed/oschaaf-multiple-experiment-cookies Experiments: fix sending out multiple experiment cookies	2014-01-24 13:12:27 -08:00
Otto van der Schaaf	625e762961	Experiments: fix sending out multiple experiment cookies Only classify people into an experiment when we are rewriting html. Fixes https://github.com/pagespeed/ngx_pagespeed/issues/586	2014-01-24 22:09:31 +01:00
Jeff Kaufman	c20affe323	Merge pull request #605 from pagespeed/oschaaf-date-header Date header: use current date when we don't get one handed over	2014-01-24 08:47:00 -08:00
Otto van der Schaaf	7a9e6de802	Date header: use current date when we don't get one handed over When the content generator does not supply us with a date header, we need to create one ourselves and set it to the current date. Fixes: https://github.com/pagespeed/ngx_pagespeed/issues/604 (duplicate) https://github.com/pagespeed/ngx_pagespeed/issues/577	2014-01-24 16:49:37 +01:00
Huibao Lin	96cf9a22f7	Update to 1.7.30.3 release	2014-01-16 18:37:36 -05:00
Jeff Kaufman	ab83a70a35	Merge pull request #599 from eezis/docfix Added a missing 'cd ~' command to the README	2014-01-16 13:30:44 -08:00
Ernest Ezis	658b2cf7a9	Added the missing 'cd ~' command to the '3. Download and build nginx:' section	2014-01-16 12:45:14 -07:00
Jeff Kaufman	092bbf2862	Merge pull request #597 from pagespeed/jefftk-update-trunk-tracking trunk-tracking: update from r3696 to r3714	2014-01-16 07:18:31 -08:00
Jeff Kaufman	f04c533df0	trunk-tracking: update from r3696 to r3714 https://code.google.com/p/modpagespeed/source/detail?r=3697 * system test improvements https://code.google.com/p/modpagespeed/source/detail?r=3699 * moved some config from location block to server block * system test improvements https://code.google.com/p/modpagespeed/source/detail?r=3707 * tests for OptimizeForBandwidth * had to switch tests from directory blocks to server+location blocks https://code.google.com/p/modpagespeed/source/detail?r=3708 * update to a test that had never been ported * ngx_pagespeed.cc: * change in signature of FindIgnoreCase https://code.google.com/p/modpagespeed/source/detail?r=3689 * Was apparently skipped with #591.	2014-01-15 13:18:47 -05:00
Jeff Kaufman	8468e4849a	Merge branch 'master' into trunk-tracking	2014-01-08 10:51:42 -05:00
Jeff Kaufman	df5736609d	native-fetcher: add support for FetchProxy The native fetcher previously ignored FetchProxy settings; now it doesn't. Squash-merge of tcpper's #590.	2014-01-08 10:51:24 -05:00
Jeff Kaufman	7fbb2c61ee	readme: release 1.7.30.2	2014-01-06 16:51:41 -05:00
Jeff Kaufman	af772c2fe8	Merge pull request #592 from pagespeed/jefftk-better-configure-error config: point people to obj/autoconf.err when psol isn't detected	2014-01-03 03:33:47 -08:00
Jeff Kaufman	a4bd9b9c13	config: point people to obj/autoconf.err when psol isn't detected by ./configure	2014-01-02 23:09:42 -05:00
Jeff Kaufman	f87d0f7ae2	Merge pull request #591 from pagespeed/jud-trunk-tracking Updates to trunk tracking branch.	2014-01-02 13:43:11 -08:00
Jud Porter	501742cb56	trunk-tracking: update from r3677 to r3696. Add inline unauthorized resources test and fix rendered image dimensions test.	2014-01-02 16:42:10 -05:00
Jeff Kaufman	a669be99b1	Merge branch 'master' into trunk-tracking	2014-01-02 16:25:23 -05:00
Jeff Kaufman	328d3afc9b	Merge pull request #583 from pagespeed/jefftk-support-purge native-fetcher: support non-GET request methods like PURGE	2013-12-20 07:11:55 -08:00
Jeff Kaufman	2681c24ee0	native-fetcher: fix to work with nginx 1.5.8+ nginx 1.5.8 changed the resolver api, which the native fetcher uses. Fixes #578. Squash-merge of @dinic's #581.	2013-12-19 12:46:53 -05:00
Jeff Kaufman	f86f47fda4	native-fetcher: support non-GET request methods like PURGE	2013-12-19 11:38:46 -05:00
Jeff Kaufman	179c81afa3	test: don't run downstream caching test with native fetcher	2013-12-19 11:03:53 -05:00
Jeff Kaufman	8d7eb20c89	Merge pull request #575 from pagespeed/dec9-trunk-tracking-updates Trunk tracking updates to sync to PSOl r3677.	2013-12-13 07:14:30 -08:00
Anupama Dutta	1fe6c54b94	Trunk tracking updates to sync to PSOl r3677. Includes minor additions to test only. Trunk tracking updates to sync to PSOl r3677. Trunk tracking update to sync to PSOL r3677 Includes minor additions to test only.	2013-12-12 22:27:27 -05:00
Jeff Kaufman	1667879202	Merge pull request #571 from morlovich/morlovich-trunk-tracking-20131125 Trunk tracking update for up to r3646	2013-12-02 07:45:08 -08:00
Maks Orlovich	0076e45677	Port over ModPagespeed r3635: RespectXForwardedProto is vhost-scope	2013-11-27 12:59:47 -05:00
Jeff Kaufman	1f3560ea21	backport header-only fix Was: trunk-tracking: update to r3632 from 1.7.30.1	2013-11-26 16:39:52 -05:00
Jeff Kaufman	54352bff72	trunk-tracking: update to r3632 from 1.7.30.1	2013-11-26 16:37:30 -05:00
Maks Orlovich	329985659c	Fix flakey test It doesn't make sense to fetch_until the uncombined URL and then grep for combined one: you might just get entirely unoptimized output and fail the test. Instead wait for the combining to happen, and make sure it combined the right # of things.	2013-11-25 16:32:31 -05:00
Maks Orlovich	db870f7023	Update to pagespeed_automatic.ca more symbols + symbol renaming This make it possible to build w/o disabling SSL, makes us need less extra .a + .cc files, but does mean that APR can't be used directly (since it got renamed), so just use PosixTimer where it was used.	2013-11-25 15:36:30 -05:00
Jeff Kaufman	ed14455412	valgrind: unflake cache purging test Fixes #569.	2013-11-25 14:38:24 -05:00
Jeff Kaufman	be4d263d10	valgrind: suppressions file might not be in current directory	2013-11-25 10:23:25 -05:00
Jeff Kaufman	0bafd6b7e0	Merge pull request #565 from pagespeed/oschaaf-valgrind Valgrind: Add an automated test	2013-11-25 07:03:33 -08:00
Otto van der Schaaf	9bbe912bd7	Valgrind: Add an automated test This makes nginx run in the background under valgrind, with both a master and a child process. Valgrind errors will be redirected to `valgrind.log`. When `USE_VALGRIND` is set, all system tests will be run under valgrind, and at the end a new test is appended which ensures no valgrind errors where encountered. It is also worth noting that: - There is a new file, `valgrind.sup`, which contains a few suppressions. - Some tests behave flakey under valgrind. For now these are appended to the expected failures (when under valgrind only). - 'Possibly lost' errors are all suppressed to get the amount of false positives manageable.	2013-11-21 21:26:15 +01:00
Jeff Kaufman	b78eb8a939	Merge pull request #567 from pagespeed/oschaaf-304-timeout system-tests: Test keepalive behaviour after a 304 response	2013-11-21 11:05:18 -08:00
Otto van der Schaaf	e082a01912	system-tests: Test keepalive behaviour after a 304 response	2013-11-20 23:15:14 +01:00
Jeff Kaufman	fa5815e1e8	Merge pull request #560 from pagespeed/jefftk-fix-messages messages: unbreak /ngx_pagespeed_messages	2013-11-12 11:51:57 -08:00
Jeff Kaufman	f12af2f03b	messages: unbreak /ngx_pagespeed_messages The shared circular buffer wasn't hooked up fully, which meant loading /ngx_pagespeed_messages didn't work. This fixes that and adds a test. I also noticed while adding this that the 'Handling of large files' test wasn't set up properly, so I converted that to use start_test. Fixing that exposed another bug where the 'Handling of large files' test was actually failing but being marked as an expected failure by being grouped in with the test above. Adding `pagespeed MaxHtmlParseBytes 5000` to the appropriate location made it test what it was supposed to be testing again, and the underlying feature wasn't broken.	2013-11-12 13:11:12 -05:00
Jeff Kaufman	e22fae46bc	readme: use release	2013-11-08 11:41:53 -05:00