Metadata-Version: 2.1
Name: scrapy-fake-useragent
Version: 1.4.3
Summary: Use a random User-Agent provided by fake-useragent for every request
Home-page: https://github.com/alecxe/scrapy-fake-useragent
Author: Alexander Afanasyev
Author-email: me@alecxe.me
Maintainer: Alexander Afanasyev
Maintainer-email: me@alecxe.me
License: New BSD License
Keywords: scrapy proxy user-agent web-scraping
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: Scrapy
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.5
Requires-Dist: fake-useragent
Requires-Dist: faker

.. image:: https://travis-ci.org/alecxe/scrapy-fake-useragent.svg?branch=master
    :target: https://travis-ci.org/alecxe/scrapy-fake-useragent

.. image:: https://codecov.io/gh/alecxe/scrapy-fake-useragent/branch/master/graph/badge.svg
  :target: https://codecov.io/gh/alecxe/scrapy-fake-useragent

.. image:: https://img.shields.io/pypi/pyversions/scrapy-fake-useragent.svg
     :target: https://pypi.python.org/pypi/scrapy-fake-useragent
     :alt: PyPI version

.. image:: https://badge.fury.io/py/scrapy-fake-useragent.svg
     :target: http://badge.fury.io/py/scrapy-fake-useragent
     :alt: PyPI version

.. image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master
     :target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=master
     :alt: Requirements Status

.. image:: https://img.shields.io/badge/license-MIT-blue.svg
     :target: https://github.com/alecxe/scrapy-fake-useragent/blob/master/LICENSE.txt
     :alt: Package license

scrapy-fake-useragent
=====================

Random User-Agent middleware for Scrapy scraping framework based on
`fake-useragent <https://pypi.python.org/pypi/fake-useragent>`__, which picks up ``User-Agent`` strings 
based on `usage statistics <http://www.w3schools.com/browsers/browsers_stats.asp>`__
from a `real world database <http://useragentstring.com/>`__, but also has the option to configure a generator
of fake UA strings, as a backup, powered by 
`Faker <https://faker.readthedocs.io/en/stable/providers/faker.providers.user_agent.html>`__.

It also has the possibility of extending the
capabilities of the middleware, by adding your own providers.

Changes
----------

Please see `CHANGELOG`_.

Installation
-------------

The simplest way is to install it via `pip`:

    pip install scrapy-fake-useragent

Configuration
-------------

Turn off the built-in ``UserAgentMiddleware`` and ``RetryMiddleware`` and add
``RandomUserAgentMiddleware`` and ``RetryUserAgentMiddleware``.

In Scrapy >=1.0:

.. code:: python

    DOWNLOADER_MIDDLEWARES = {
        'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
        'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
        'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
        'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,
    }

In Scrapy <1.0:

.. code:: python

    DOWNLOADER_MIDDLEWARES = {
        'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
        'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': None,
        'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
        'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,
    }

Recommended setting (1.3.0+):

.. code:: python

    FAKEUSERAGENT_PROVIDERS = [
        'scrapy_fake_useragent.providers.FakeUserAgentProvider',  # this is the first provider we'll try
        'scrapy_fake_useragent.providers.FakerProvider',  # if FakeUserAgentProvider fails, we'll use faker to generate a user-agent string for us
        'scrapy_fake_useragent.providers.FixedUserAgentProvider',  # fall back to USER_AGENT value
    ]
    USER_AGENT = '<your user agent string which you will fall back to if all other providers fail>'

----------------

Additional configuration information
====================================

Enabling providers
---------------------------

The package comes with a thin abstraction layer of User-Agent providers, which for purposes of backwards compatibility defaults to:

.. code:: python

    FAKEUSERAGENT_PROVIDERS = [
        'scrapy_fake_useragent.providers.FakeUserAgentProvider'
    ]

The package has also ``FakerProvider`` (powered by `Faker library <https://faker.readthedocs.io/>`__) and ``FixedUserAgentProvider`` implemented and available for use if needed.

Each provider is enabled individually, and used in the order they are defined.
In case a provider fails execute (for instance, it can `happen <https://github.com/hellysmile/fake-useragent/issues/99>`__ to fake-useragent because of it's dependency
with an online service), the next one will be used.

Example of what ``FAKEUSERAGENT_PROVIDERS`` setting may look like in your case:

.. code:: python

    FAKEUSERAGENT_PROVIDERS = [
        'scrapy_fake_useragent.providers.FakeUserAgentProvider',
        'scrapy_fake_useragent.providers.FakerProvider',
        'scrapy_fake_useragent.providers.FixedUserAgentProvider',
        'mypackage.providers.CustomProvider'
    ]


Configuring fake-useragent
---------------------------

Parameter: ``FAKE_USERAGENT_RANDOM_UA_TYPE`` defaulting to ``random``.

Other options, as example: 

* ``firefox`` to mimic only firefox browsers
* ``desktop`` or ``mobile`` values to send desktop or mobile strings respectively.

You can also set the ``FAKEUSERAGENT_FALLBACK`` option, which is a ``fake-useragent`` specific fallback. For example:

.. code:: python

    FAKEUSERAGENT_FALLBACK = 'Mozilla/5.0 (Android; Mobile; rv:40.0)'

What it does is, if the selected ``FAKE_USERAGENT_RANDOM_UA_TYPE`` fails to retrieve a UA, it will use
the type set in ``FAKEUSERAGENT_FALLBACK``.

Configuring faker
---------------------------

Parameter: ``FAKER_RANDOM_UA_TYPE`` defaulting to ``user_agent`` which is the way of selecting totally random User-Agents values.
Other options, as example:

* ``chrome``
* ``firefox``

Configuring FixedUserAgent
---------------------------

It also comes with a fixed provider (only provides one user agent), reusing the Scrapy's default ``USER_AGENT`` setting value.

Usage with `scrapy-proxies`
---------------------------

To use with middlewares of random proxy such as `scrapy-proxies <https://github.com/aivarsk/scrapy-proxies>`_, you need:

1. set ``RANDOM_UA_PER_PROXY`` to True to allow switch per proxy

2. set priority of ``RandomUserAgentMiddleware`` to be greater than ``scrapy-proxies``, so that proxy is set before handle UA

License
----------

The package is under MIT license. Please see `LICENSE`_.

.. |GitHub version| image:: https://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent.svg
   :target: http://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent
.. |Requirements Status| image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master
   :target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=master
.. _LICENSE: https://github.com/alecxe/scrapy-fake-useragent/blob/master/LICENSE.txt
.. _CHANGELOG: https://github.com/alecxe/scrapy-fake-useragent/blob/master/CHANGELOG.rst


