* respect purge requests when serving ipro requests from ngx_pagespeed
* Add TODO to share common base with ApacheFetch.
During system-tests, don't write into the source-controlled htdocs area. (#1197)
Instead, use a new (not-yet-committed) target in Makefile.tests to build
a mirror htdocs in test/tmp/root that sym-links the large readonly
section and deep copies the test areas for purge and flush tests.
Pass the install-path into Makefile (#1200)
* Pass the install-path into Makefile, rather than having the Makefile
depend on PWD.
* Move the double-quotes be before the value, to be consistent with other
Makefiles.
- Fix valgrind errors
- Add a test to make sure all logged output looks sane by whitelisting
current errors/warnings.
- Stop our nginx test instances after we are done testing.
- Add tests for shutting down and reloading configuration under high
load (depends on ab).
- Reduce the number of keepalive requests in the keepalive tests to speed
up test runs.
- Fix exiting with open file descriptors, fix cleanup in nginx's cache
manager/loader processes
- Attempt to finish up queued up NgxBaseFetches/requests on shutdown/reload
- Under valgrind the blocking rewrite started failing after adding a test
for reloading configuration under high load.
I've added it to the expected failures for valgrind, looking into this
is up next.
- Decline in ps_resource_handler when nginx is quitting. This makes us
more reliable on continued stress during shutdown/reload.
This change passes on non succesful status codes for pagespeed
resources and other places where we act as a content handler to
nginx. This has two benefits:
- Instead of a blank page, the user agent receives a formatted
and (hopefully customized and informative) response.
- Header modules are able to operate on that response, which was
requested in https://github.com/pagespeed/ngx_pagespeed/issues/612#issuecomment-58855816
- Don't construct `NgxBaseFetch` instances and call `Done(false)` later when it turns out it wasn't needed.
- Track the purpose of the base fetch instead of just wether it is an IPRO lookup or not for more informational tracing
as well as avoiding constructing base fetches when their purpose isn't know yet.
- Don't initiate a property cache lookup when we won't need it (`ProxyFetchFactory::InitiatePropertyCacheLookup`)
- Improve IPRO handling of `NgxBaseFetch` and make it clearer. Indicate we want to supress further events when the
base fetch was used to lookup an IPRO url, and the result was not indicative of something we can respond with to the
browser.
- Remove unused `RequestRouting` constants
Abstract the pipe communication into NgxEventConnection, for reuse
by NgxBaseFetch and NgxUrlAsyncFetcher.
Based on Chai's earlier work, but with a few fixes discovered
while working on this and SPDY module compatibility
- Uses less file descriptors, I expect this to be faster but need
measurement is needed to back that.
- Fixed NgxUrlAsyncFetcher actually shutting down its fetchers.
- Fixes a bug where we wouldn't clean idle pooled NgxConnections.
- Fixes a bug for requests that are finalized mid-IPRO lookup.
- Makes us use ngx_handle_read_event/ngx_del_event
For FetchInPlaceResource, NgxBaseFetch would send two bytes down its
pipe, one upon HeaderComplete() and one upon HandleDone(). We need
only one to resume processing on the nginx side.
There is a race between ps_connection_read_handler() and processing
of the byte send by NgxBaseFetch::HandleDone().
ps_connection_read_handler() clears the pipe when the request is
finalized, and also drains it on each event - so two writes could be
processed as one when lucky, masking the problem).
One concrete problem this solved for me was that SPDY + IPRO +
proxy_pass would segfault, hang, and/or pass on 5xx/404 responses
from IPRO lookup fetches to the browser, next to alerts about
r->count being zero in nginx's error.log
Might fix https://github.com/pagespeed/ngx_pagespeed/issues/788
Fixes https://github.com/pagespeed/ngx_pagespeed/issues/792
Changes:
* r3585: With downstream caching, don't touch `Cache-Control` headers.
* No longer require `Modify Caching Headers off`.
* Change from `modify_caching_headers` as a boolean to a three valued enum
`PreserveCachingHeaders`.
* r3596: Make tests less flaky.
* Changes WGET_DUMP to write to WGET_DIR instead of OUTDIR.
* r3597: Remove now-redundant system tests.
* Some system tests were moved into `automatic/system_test.sh` which means we
can remove our forks.
* r3598: Enable the shared memory metadata cache by default.
* The code change just worked and needed no changes, but it also added
substantial tests, which needed some porting.
* Deflake the `scrape stats` test by moving it before config reloading.
* Remove PopulateRequestHeaders() and PopulateResponseHeaders().
* Set stop copying request headers to the base fetch twice.
* This fixes a dcheck failure in the debug build.
Squash-merge of chaizhenhua's work over many pull requests, especially #450.
The copy of in_place_resource_recorder.cc is temporary and can be removed after
the next PSOL release.
* Add tests for statistics.
* We weren't increasing resource_404_count on 404s.
* This required giving `NgxBaseFetch` a `NgxServerContext` pointer.
* /ngx_pagespeed_statistics was publically available.
* Made a pass over the readme to add a new configuration option and fixed up
serveral things that were out of date while I was there.
Fixes#248.
This adds /ngx_pagespeed_statistics. It also uses SharedMemoryStatistics instead
of the SimpleStatistics currently used, and sets them as the default. It also
adds support for tracking per server{} statistics.
Cleaning up of shared segments is disabled, because the current psol version we
link in has a problem with the ordering of nginx's events during a configuration
reload.
The referrer statistics will be ported in a separate pull.
Related to issue #133
Squash-merge of Otto's work in #176.
When a request is cancelled part way through nginx calls the cleanup
handler. This was set to delete the base fetch, but the proxy fetch was
sometimes not done with it yet. Both nginx and pagespeed must be finished with
the base fetch before it's safe to delete it. Pagespeed calls Done(), and this
adds Release() for nginx to call. Whichever is called second deletes the
base_fetch. We use a count of references that starts at two and is decremented
atomically with a memory barrier once when Done() is called and once when
Release() is called.
Fixes#103, #104, and #106.
The pagespeed api for handling logs has changed, we need to change with it in
order to build against the latest pagespeed optimization libraries.
Separately, a new test was added to the generic system test that depends on the
property cache (insert_dns_prefetch). We've not ported the property cache yet,
so this test will fail, and I've added it to the list of failing tests.
Pagespeed generates urls like:
/script.js.pagespeed.ce.FjFFoSiQs4.js
when it moves a resource. Add a content handler that accepts
these requests and passes them through to pagespeed.
Jeffrey Crowell