Matt Howlett

Matt Howlett

every blog should have a tagline

PhantomJS - Convincingly Faking Referrer Behavior


Faking a referring page convincingly with PhantomJS [version 1.9] takes a bit of work.

It is possible to set the page.customHeaders property, however:

  1. The javascript document.referrer property in the fetched page does not get correctly set.
  2. The referrer is applied to all HTTP requests (including embedded images, scripts etc), not just the initial request which is the standard browser behavior.

Others have suggested:

  1. Open the desired referrer page.
  2. Inject a link in the retrieved page pointing to the page required.
  3. Programatically click the link.

The two problems with this are:

  1. It's wasted resources to request the referring page as well as the desired page. It's easy to do better here - PhantomJS provides a setContent method which allows the page content and the corresponding (fake) url to be set - no need to actually load the referring page, just use this method. Include a link to the desired page, and programatically click it.
  2. For some reason, a custom user agent string applied via page.settings.userAgent is ignored when programatically clicking on a link (it only seems to work with The easiest way to resolve this (that I could think of) is to hack the userAgentForUrl method in the PhantomJS source here: src/qt/src/3rdparty/webkit/Source/WebKit/qt/Api/qwebpage.cpp and recompile. I.e. set the default user agent string to what you want. This can still be overridden in js in the usual way of course for the cases where PhantomJS behaves as might be expected.

Now referring page behavior is indistinguishable from that of a manual user.