Metadata-Version: 1.1
Name: scrapy-pipeline-mongodb
Version: 0.0.4
Summary: UNKNOWN
Home-page: https://github.com/grammy-jiang/scrapy-pipeline-mongodb
Author: Grammy Jiang
Author-email: grammy.jiang@gmail.com
License: BSD
Description-Content-Type: UNKNOWN
Description: =======================
        Scrapy-Pipeline-MongoDB
        =======================
        
        .. image:: https://img.shields.io/pypi/v/scrapy-pipeline-mongodb.svg
           :target: https://pypi.python.org/pypi/scrapy-pipeline-mongodb
           :alt: PyPI Version
        
        .. image:: https://img.shields.io/travis/grammy-jiang/scrapy-pipeline-mongodb/master.svg
           :target: http://travis-ci.org/grammy-jiang/scrapy-pipeline-mongodb
           :alt: Build Status
        
        .. image:: https://img.shields.io/badge/wheel-yes-brightgreen.svg
           :target: https://pypi.python.org/pypi/scrapy-pipeline-mongodb
           :alt: Wheel Status
        
        .. image:: https://img.shields.io/codecov/c/github/grammy-jiang/scrapy-pipeline-mongodb/master.svg
           :target: http://codecov.io/github/grammy-jiang/scrapy-pipeline-mongodb?branch=master
           :alt: Coverage report
        
        Overview
        ========
        
        Scrapy is a great framework for web crawling. This package provides two
        pipelines to save items into MongoDB in a async or sync way. And also provide a
        a highly customized way to interact with MongoDB in a async or sync way.
        
        * Save an item and get Object ID from this pipeline
        
        * Update an item and get Object ID from this pipeline
        
        Requirements
        ============
        
        * Txmongo, a async MongoDB driver with Twisted
        
        * Not support Python 2.7
        
        * Tests on Python 3.5, but it should work on other version higher then Python
          3.3
        
        * Tests on Linux, but it's a pure python module, it should work on other
          platforms with official python and Twisted supported, e.g. Windows, Mac OSX,
          BSD
        
        Installation
        ============
        
        The quick way::
        
            pip install scrapy-pipeline-mongodb
        
        Or put this middleware just beside the scrapy project.
        
        Documentation
        =============
        
        Block Inspector in spider middleware, in ``settings.py``, for example::
        
            from txmongo.filter import ASCENDING
            from txmongo.filter import DESCENDING
        
            # -----------------------------------------------------------------------------
            # PIPELINE MONGODB ASYNC
            # -----------------------------------------------------------------------------
        
            ITEM_PIPELINES.update({
                'scrapy_pipeline_mongodb.pipelines.mongodb_async.PipelineMongoDBAsync': 500,
            })
        
            MONGODB_USERNAME = 'user'
            MONGODB_PASSWORD = 'password'
            MONGODB_HOST = 'localhost'
            MONGODB_PORT = 27017
            MONGODB_DATABASE = 'test_mongodb_async_db'
            MONGODB_COLLECTION = 'test_mongodb_async_coll'
        
            # MONGODB_OPTIONS_ = 'MONGODB_OPTIONS_'
        
            MONGODB_INDEXES = [('field_0', ASCENDING, {'unique': True}),
                               (('field_0', 'field_1'), ASCENDING),
                               (('field_0', ASCENDING), ('field_0', DESCENDING))]
        
            MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'
        
        
        Settings Reference
        ==================
        
        MONGODB_USERNAME
        ----------------
        
        A string of the username of the database.
        
        MONGODB_PASSWORD
        ----------------
        
        A string of the password of the database.
        
        MONGODB_HOST
        ------------
        
        A string of the ip address or the domain of the database.
        
        MONGODB_PORT
        ------------
        
        A int of the port of the database.
        
        MONGODB_DATABASE
        ----------------
        
        A string of the name of the database.
        
        MONGODB_COLLECTION
        ------------------
        
        A list of the indexes to create on the collection.
        
        MONGODB_OPTIONS_*
        -----------------
        
        Options can be attached when the pipeline start to connect to MongoBD.
        
        If any options are needed, the name of the option can be with the prefix
        `MONGODB_OPTIONS_`, the pipeline will parse it.
        
        For example:
        
        +---------------+-------------------------------+
        | option name   | in `settings.py`              |
        +---------------+-------------------------------+
        | authMechanism | MONGODB_OPTIONS_authMechanism |
        +---------------+-------------------------------+
        
        
        For more options, please refer to the page:
        
        `Connection String URI Format — MongoDB Manual 3.4`_
        
        .. _`Connection String URI Format — MongoDB Manual 3.4`: https://docs.mongodb.com/manual/reference/connection-string/#connections-standard-connection-string-format
        
        MONGODB_INDEXES
        ---------------
        
        A list of the indexes defined in this setting will be created when the spider is
        open.
        
        If the index has already existed, there will be no warning or error raised.
        
        MONGODB_PROCESS_ITEM
        --------------------
        
        To highly customize to interact with MongoDB, this pipeline provide a setting to
        define the function `process_item`. And with this package, there is one default
        function: just call the method `insert_one` of the collection to save the item
        into MongoDB, then return the item.
        
        If a customize is provided to replace the default one, please note the behavior
        should follow the requirement which is clearly written in the scrapy documents:
        
        `Item Pipeline — Scrapy 1.4.0 documentation`_
        
        .. _`Item Pipeline — Scrapy 1.4.0 documentation`: https://doc.scrapy.org/en/latest/topics/item-pipeline.html#writing-your-own-item-pipelin
        
        Build-in Functions For Processing Item
        ======================================
        
        scrapy_pipeline_mongodb.utils.process_item.process_item
        -------------------------------------------------------
        
        This is a build-in function to call the method `insert_one` of the collection,
        and return the item.
        
        To use this function, in `settings.py`::
        
            MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'
        
        NOTE
        ====
        
        The drivers may have different api for the same operation, this pipeline adopts
        txmongo as the async driver for MongoDB, please read the relative documents to
        make sure the customized functions can run fluently in this pipeline.
        
        TODO
        ====
        * Add a unit test for the index created function
        
        * Add a sync pipeline
        
Platform: UNKNOWN
Classifier: Framework :: Scrapy
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Plugins
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
