Metadata-Version: 2.1
Name: search_engines
Version: 1.0.7
Summary: Query and scrape search engines.
Home-page: https://github.com/djkelleher/search_engines
Author: Dan Kelleher
Author-email: kelleherjdan@gmail.com
Maintainer: Dan Kelleher
Maintainer-email: kelleherjdan@gmail.com
License: UNKNOWN
Description: 
        ### **Query and scrape search engines (Google, Google News, Yahoo, Yahoo News, Bing, Bing News, Ask, Dogpile, Dogpile News)**
        ----
        ## Installation
        ```pip install search_engines```   
        
        ## Overview
        Each search engine has a module {engine_name}.py which two functions:   
        ```python 
        extract_search_results(html: str, page_url: str) -> Tuple[List[Dict[str, str]], str]
        ```  
        and  
        ```python
        get_search_url(query: str, latest: bool = True, country: str = 'us') -> str
        ```
        
        ## Usage Example
        Construct a URL for the first results page of searching "Tesla TSLA" in Bing Search.
        ```python
        from search_engines import bing_search
        
        url = bing_search.get_search_url('Tesla TSLA')
        ```
        Load the URL using a simple HTTP client or web browser and extract the page HTML.
        This package does not make any restrictions on clients can be used. We'll use the `requests` library for this example.
        ```python
        import requests
        
        resp = requests.get(url)
        html = resp.text
        ```
        We can now extract search results from the HTML.
        The returned `results` list will be a list of dictionaries with keys `url`, `title`, `preview_text`, `page_number`.
        If we want to scrape multiple pages, we can load the next page using the returned `next_page_url`, and again extracting the results using `extract_search_results`.
        
        ```python
        results, next_page_url = bing_search.extract_search_results(html, url)
        ```
        
        ## Contributions
        Add new search engines! =)   
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
Provides-Extra: test
