Metadata-Version: 2.1
Name: scrapy-sticky-meta-params
Version: 1.0.0
Summary: A spider middleware that forwards meta params through subsequent requests.
Home-page: https://github.com/heylouiz/scrapy-sticky-meta-params
Author: Luiz Francisco Rodrigues da Silva
Author-email: luizfrdasilva@gmail.com
License: MIT license
Keywords: scrapy meta middleware
Platform: Any
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Description-Content-Type: text/markdown
License-File: LICENSE

# scrapy-sticky-meta-params

A Scrapy spider middleware that forwards meta params through subsequent requests.

## What does it do?

This middleware simplify the process of carrying information through requests and responses on spiders.

## Without the middleware

    class SampleSpider(Spider):
        name = "without_middleware"
        start_urls = ["https://www.example.com"]

        def parse(self, response):
            for param in range(5):
                yield Request(
                    "https://www.example.com/next",
                    meta={"param": param},
                    callback=self.parse_2
                )

        def parse_2(self, response):
            # Get important information from response
            info = response.xpath("//info/text()").get("info")
            # We need to get the param from meta and forward it again in this request
            param = response.meta["param"]
            yield Request(
                "https://www.example.com/next",
                meta={"info": info, "param": param},
                callback=self.parse_3
            )

        def parse_3(self, response):
            yield {
                "param": response.meta["param"],  # The value that we've extracted in the first callback
                "info": response.meta["info"]
            }

## With the middleware

    class SampleSpider(Spider):
        name = "with_middleware"
        start_urls = ["https://www.example.com"]
        sticky_meta_keys = ["param"]  # Will always forward the meta param "param"

        def parse(self, response):
            for param in range(5):
                yield Request(
                    "https://www.example.com/next",
                    meta={"param": param},
                    callback=self.parse_2
                )

        def parse_2(self, response):
            # Get important information from response
            info = response.xpath("//info/text()").get("info")
            # We don"t need to get the "param" value from meta and resend it.
            yield Request(
                "https://www.example.com/next",
                meta={"info": info},
                callback=self.parse_3
            )

        def parse_3(self, response):
            yield {
                "param": response.meta["param"],  # The value that we've extracted in the first callback
                "info": response.meta["info"]
            }


## Awesome, how to use it?

To enable the middleware you need to add it to your projects's `SPIDER_MIDDLEWARES` setting in `settings.py`.

    SPIDER_MIDDLEWARES = {
        'scrapy_sticky_meta_params.middleware.StickyMetaParamsMiddleware': 550,
    }

This middleware needs to be enabled per spider, to do this you need to add the following attribute on your spider:

    sticky_meta_keys = []

You need to fill this list with every key that you want to be forwarded to subsequent requests.


