This change makes ngx_pagespeed listen to the FollowFlushes option.
When set to on (=default), ngx_pagespeed will forward incoming flushes
to ProxyFetch. When writing output, we'll now also set the flush flag on
the buffers we are about to send downstream.
Companion to mps commit: https://github.com/pagespeed/mod_pagespeed/commit/02de03e825bbd1f8d4ad4e1a1bef5263a16f3857
- Fix nginx-side flow so we handle .pagespeed. resources ok
when they will land on a customized 404 internal location.
- Additionally, check for a wiped request context and make sure
we do not dereference a null pointer, which is what hurt in
the flow we entered above as the IPRO lookup still was
generating events while the nginx side request context was
gone.
- Also, as a preliminary measure, do not check fail when we
receive a stale event originating from a NgxBaseFetch that
is no longer associated with our request context.
Do log a warning so we'll hear about this happening either
through system test failures or a bug report.
Fixes https://github.com/pagespeed/ngx_pagespeed/issues/1081
- Fix valgrind errors
- Add a test to make sure all logged output looks sane by whitelisting
current errors/warnings.
- Stop our nginx test instances after we are done testing.
- Add tests for shutting down and reloading configuration under high
load (depends on ab).
- Reduce the number of keepalive requests in the keepalive tests to speed
up test runs.
- Fix exiting with open file descriptors, fix cleanup in nginx's cache
manager/loader processes
- Attempt to finish up queued up NgxBaseFetches/requests on shutdown/reload
- Under valgrind the blocking rewrite started failing after adding a test
for reloading configuration under high load.
I've added it to the expected failures for valgrind, looking into this
is up next.
- Decline in ps_resource_handler when nginx is quitting. This makes us
more reliable on continued stress during shutdown/reload.
Get rid of write_pending & fetch_done flags on the request context.
Assume that having ctx->base_fetch set means we have outstanding
work and thus we are not done yet, for simplicity.
This also means we'll release the base fetch earlier which seems
like a good idea to me.
- Don't construct `NgxBaseFetch` instances and call `Done(false)` later when it turns out it wasn't needed.
- Track the purpose of the base fetch instead of just wether it is an IPRO lookup or not for more informational tracing
as well as avoiding constructing base fetches when their purpose isn't know yet.
- Don't initiate a property cache lookup when we won't need it (`ProxyFetchFactory::InitiatePropertyCacheLookup`)
- Improve IPRO handling of `NgxBaseFetch` and make it clearer. Indicate we want to supress further events when the
base fetch was used to lookup an IPRO url, and the result was not indicative of something we can respond with to the
browser.
- Remove unused `RequestRouting` constants
Request termination code relies on posted requests being run.
One of the things this fixes is that we won't hang when configured
to run before the SSI module.
Avoids trouble when changing the module order, which helps
https://github.com/pagespeed/ngx_pagespeed/issues/612
Abstract the pipe communication into NgxEventConnection, for reuse
by NgxBaseFetch and NgxUrlAsyncFetcher.
Based on Chai's earlier work, but with a few fixes discovered
while working on this and SPDY module compatibility
- Uses less file descriptors, I expect this to be faster but need
measurement is needed to back that.
- Fixed NgxUrlAsyncFetcher actually shutting down its fetchers.
- Fixes a bug where we wouldn't clean idle pooled NgxConnections.
- Fixes a bug for requests that are finalized mid-IPRO lookup.
- Makes us use ngx_handle_read_event/ngx_del_event
For FetchInPlaceResource, NgxBaseFetch would send two bytes down its
pipe, one upon HeaderComplete() and one upon HandleDone(). We need
only one to resume processing on the nginx side.
There is a race between ps_connection_read_handler() and processing
of the byte send by NgxBaseFetch::HandleDone().
ps_connection_read_handler() clears the pipe when the request is
finalized, and also drains it on each event - so two writes could be
processed as one when lucky, masking the problem).
One concrete problem this solved for me was that SPDY + IPRO +
proxy_pass would segfault, hang, and/or pass on 5xx/404 responses
from IPRO lookup fetches to the browser, next to alerts about
r->count being zero in nginx's error.log
Might fix https://github.com/pagespeed/ngx_pagespeed/issues/788
Fixes https://github.com/pagespeed/ngx_pagespeed/issues/792
Changes:
* r3585: With downstream caching, don't touch `Cache-Control` headers.
* No longer require `Modify Caching Headers off`.
* Change from `modify_caching_headers` as a boolean to a three valued enum
`PreserveCachingHeaders`.
* r3596: Make tests less flaky.
* Changes WGET_DUMP to write to WGET_DIR instead of OUTDIR.
* r3597: Remove now-redundant system tests.
* Some system tests were moved into `automatic/system_test.sh` which means we
can remove our forks.
* r3598: Enable the shared memory metadata cache by default.
* The code change just worked and needed no changes, but it also added
substantial tests, which needed some porting.
* Deflake the `scrape stats` test by moving it before config reloading.
Get rid of net_instaweb:: everywhere. By eliminating the ngx_psol namespace we
can stop prefixing everything in ngx_pagespeed.cc with net_instaweb::
Fix other minor style issues.
Remove unnecessary declarations.
Rename ps_create_request_context to ps_route_request and CreateRequestContext to
RequestRouting because it no longer creates the request context.
* Remove PopulateRequestHeaders() and PopulateResponseHeaders().
* Set stop copying request headers to the base fetch twice.
* This fixes a dcheck failure in the debug build.
Squash-merge of chaizhenhua's work over many pull requests, especially #450.
The copy of in_place_resource_recorder.cc is temporary and can be removed after
the next PSOL release.
* Add tests for statistics.
* We weren't increasing resource_404_count on 404s.
* This required giving `NgxBaseFetch` a `NgxServerContext` pointer.
* /ngx_pagespeed_statistics was publically available.
* Made a pass over the readme to add a new configuration option and fixed up
serveral things that were out of date while I was there.
Fixes#248.
This adds /ngx_pagespeed_statistics. It also uses SharedMemoryStatistics instead
of the SimpleStatistics currently used, and sets them as the default. It also
adds support for tracking per server{} statistics.
Cleaning up of shared segments is disabled, because the current psol version we
link in has a problem with the ordering of nginx's events during a configuration
reload.
The referrer statistics will be ported in a separate pull.
Related to issue #133
Squash-merge of Otto's work in #176.
When a request is cancelled part way through nginx calls the cleanup
handler. This was set to delete the base fetch, but the proxy fetch was
sometimes not done with it yet. Both nginx and pagespeed must be finished with
the base fetch before it's safe to delete it. Pagespeed calls Done(), and this
adds Release() for nginx to call. Whichever is called second deletes the
base_fetch. We use a count of references that starts at two and is decremented
atomically with a memory barrier once when Done() is called and once when
Release() is called.
Fixes#103, #104, and #106.
The pagespeed api for handling logs has changed, we need to change with it in
order to build against the latest pagespeed optimization libraries.
Separately, a new test was added to the generic system test that depends on the
property cache (insert_dns_prefetch). We've not ported the property cache yet,
so this test will fail, and I've added it to the list of failing tests.
We were sometimes failing to set last_buf when we should, but being protected
from the consequences by another bug that made us usually send additional extra
buffers with last_buf set.
See discussion in #78, pull request was #94.
Within the anonymous namespace in ngx_pagespeed.css we don't really need to be
using the long `ngx_http_pagespeed_` prefix. Switching to just `ps_` should be
safe. For symbols that need to be visible outside this file, though, we need to
keep the long prefix.
Fixes#85.
Requests to:
/ngx_pagespeed_static/...
can be handled synchronously by pagespeed. They're used for support files for
optimizations, like js_defer.js.
This also includes a couple refactors:
- I pulled most of NgxBaseFetch::CopyBufferToNginx() out into
ngx_http_pagespeed_string_piece_to_buffer_chain().
- ngx_http_pagespeed_create_request_context() now returns a
CreateRequestContext::Response which gives callers more detailed information
about the request.
- I also pulled ngx_http_pagespeed_set_cache_control() out of
ngx_http_pagespeed_header_filter().
Fixes issue #62.
Pagespeed puts headers in ngx_base_fetch->response_headers() and nginx calls
NgxBaseFetch::CollectHeaders to get them out, which wasn't implemented before.
Other minor changes in this commit:
- whitespace fixes
- ordering of includes
- ngx_http_pagespeed_str_to_string_piece now takes its argument by value
Pagespeed generates urls like:
/script.js.pagespeed.ce.FjFFoSiQs4.js
when it moves a resource. Add a content handler that accepts
these requests and passes them through to pagespeed.
Otto van der Schaaf