Metadata-Version: 2.1
Name: cobrseo
Version: 0.0.4
Summary: A small example package
Author: Vlad Zozulia
Author-email: vlad@cobr.info
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown

# cobrseo

[![Build Status](https://img.shields.io/badge/pypi-v0.0.4-blue)](https://pypi.org/project/cobrseo/)

### How to install

```sh
pip install cobrseo==0.0.4
```

### Package structure with all methods

```
   cobrseo
   └───json
   │   └───serp_processing
   │        │   read_json()
   │        │   get_keyword()
   │        │   get_urls_by_item_type()
   │        │   get_organic_info()
   │        │   get_related_searches()
   │        │   get_people_also_ask()
   │        │   get_knowledge_graph()
   │        │   get_featured_snippet()
   │
   └───crawler
   │   └───crawler3k
   │        │   crawl_article()
   │        │   get_content_from_urls()
   |
   └───api
       └───dataforseo_organic
            │   save_dataforseo_organic_serps()
```

# **Examples**

## SERP PROCESSING

### Get organic items with the most important information:

`get_organic_info(json_path: str) -> dict`

```python
>>> from cobrseo.json.serp_processing import get_organic_info
>>> json_path = './007b1216b666d5dbe4b1b00a3b760eb4.json'

>>> get_organic_info(json_path)
{0: {'domain': 'usa.kaspersky.com', 'title': 'Your mobile security & privacy covered - Kaspersky', 'url': 'https://usa.kaspersky.com/android-security', 'description': 'Antivirus. Protects you from viruses and malware on your Android devices by detecting, isolating and removing threats · Automatic scan. Continuously scans for\xa0...', 'date': None},
1: {'domain': 'play.google.com', 'title': 'Kaspersky Security & VPN - Apps on Google Play', 'url': 'https://play.google.com/store/apps/details?id=com.kms.free&hl=en_US&gl=US', 'description': 'Free antivirus and phone security for Android™ devices from Kaspersky Kaspersky Security & VPN for Android is a FREE-to-download antivirus solution that\xa0...', 'date': None},
...
7: {'domain': 'www.pcmag.com', 'title': 'The Best Android Antivirus Apps for 2022 | PCMag', 'url': 'https://www.pcmag.com/picks/the-best-android-antivirus-apps', 'description': 'Kaspersky Internet Security includes a comprehensive Android security suite. It scans for malware on demand and in real time, and keeps you from visiting\xa0...', 'date': None}}
```

### Get all organic urls. You can specify domains, that should not be included:

`get_urls_by_item_type(json_path: str, item_type: str, url_stoplist: List[str]=['google.com','facebook.com','instagram.com']) -> List[str]:`

- Available item types: `'organic'` and `'news_search'`

```python
>>> from cobrseo.json.serp_processing import get_urls_by_item_type

>>> get_urls_by_item_type(json_path, 'organic')
['https://usa.kaspersky.com/android-security',
'https://www.tomsguide.com/reviews/kaspersky-mobile-security',
'https://kaspersky-mobile-security.en.uptodown.com/android',
'https://apps.apple.com/us/app/kaspersky-security-vpn/id1089969624',
'https://www.safetydetectives.com/best-antivirus/kaspersky/',
'https://ltonlinestore.com/1-Device-1-Year-Kaspersky-internet-Security-For-Android-p73383495',
'https://www.pcmag.com/picks/the-best-android-antivirus-apps']

>>> get_urls_by_item_type(json_path, 'organic', url_stoplist=['kaspersky.com'])
['https://play.google.com/store/apps/details?id=com.kms.free&hl=en_US&gl=US',
'https://www.tomsguide.com/reviews/kaspersky-mobile-security',
'https://kaspersky-mobile-security.en.uptodown.com/android',
'https://apps.apple.com/us/app/kaspersky-security-vpn/id1089969624',
'https://www.safetydetectives.com/best-antivirus/kaspersky/',
'https://ltonlinestore.com/1-Device-1-Year-Kaspersky-internet-Security-For-Android-p73383495',
'https://www.pcmag.com/picks/the-best-android-antivirus-apps']
```

### Get keyword from json-serp:

`get_keyword(json_path: str) -> str`

```python
>>> from cobrseo.json.serp_processing import get_keyword

>>> get_keyword(json_path)
'kaspersky mobile antivirus'
```

### Get related searches:

`get_related_searches(json_path: str) -> List[str]:`

```python
>>> from cobrseo.json.serp_processing import get_related_searches

>>> get_related_searches(json_path)
['kaspersky mobile antivirus free',
'kaspersky mobile antivirus cracked apk',
'kaspersky mobile antivirus apk',
'kaspersky mobile security android',
'kaspersky mobile antivirus download',
'kaspersky mobile security activation key',
'kaspersky free antivirus',
'kaspersky mobile antivirus review']
```

### Get people aslo ask:

`get_people_also_ask(json_path: str)-> dict:`

```python
>>> from cobrseo.json.serp_processing import get_people_also_ask

>>> get_people_also_ask(json_path)
{'questions': ['Is Kaspersky antivirus good for mobile?', 'Is Kaspersky free for mobile?', 'Which antivirus is best for mobile?', 'Do I need Kaspersky on my Android?'],
'urls': ['https://www.pcmag.com/reviews/kaspersky-internet-security-for-android', 'https://www.safetydetectives.com/blog/best-really-free-antivirus-programs-for-android/', 'https://www.tomsguide.com/best-picks/best-android-antivirus', 'https://support.kaspersky.com/consumer/products/Kaspersky_Internet_Security_for_Android'],
'descriptions': ["The Bottom Line. Kaspersky Internet Security offers Android users top-tier malware protection, great anti-phishing protection, and tools to secure and recover lost and stolen phones. But some features didn't work as advertised in our hands-on testing. Sep 30, 2015", "Kaspersky Security Free — Easy to Use with Decent On-Demand Virus Scanning. Kaspersky Security Free is a decent free internet security app for Android users — and because it only provides a couple of free features, it's very easy to use.", 'Bitdefender Mobile Security. Best paid option. ...\nNorton Mobile Security. Specifications. ...\nAvast Mobile Security. Specifications. ...\nKaspersky Mobile Antivirus. Specifications. ...\nLookout Security & Antivirus. Specifications. ...\nMcAfee Mobile Security. Specifications. ...\nGoogle Play Protect. Specifications.', 'Kaspersky Internet Security for Android provides comprehensive protection for your mobile devices. Along with providing protection against viruses and other malware, the app protects your internet connection, the data on your device, access to other apps, and also allows you to block unwanted calls.']}
```

## CRAWLER

### Crawling list of urls:

`get_content_from_urls`
Parameters:

- `urls: List[str]`: Urls to be crawled.
- `lang: List[str]=['en']`: Selected languages.
- `words_limit: tuple=(0,10000)`: Minimum and maximum word limit for article length.
- `json_path: str='file'`: Name of json file with SERP for logging purpose.

Returns:

- `List[str]`: List of crawled urls.

```python
>>> from cobrseo.crawler.crawler3k import get_content_from_urls

>>> urls = ['https://www.pcmag.com/reviews/kaspersky-internet-security-for-android',
'https://www.safetydetectives.com/blog/best-really-free-antivirus-programs-for-android/',
'https://www.tomsguide.com/best-picks/best-android-antivirus',
'https://support.kaspersky.com/consumer/products/Kaspersky_Internet_Security_for_Android']

>>> len(get_content_from_urls(urls))
4
```

## API

### DataForSeo (google organic):

`save_dataforseo_organic_serps`
Parameters:

- `keywords: List[str]`: Keywords for search.
- `destination_path: str`: Directory for json saving.
- `rewrite_serp: bool`: Allow to rewrite already saved json.
- `token: str`: API-KEY from DataForSeo.
- `post_size: int=80`: Number of keywords in one POST request (max=100, but 80 is recommended).
- `lang: str='en'`: Language (DataForSeo parameter).
- `loc: int=2840`: Location (DataForSeo parameter).
- `depth: int=10`: SERP depth (DataForSeo parameter).

Returns:

- `List[dict]`: List of dict with keywords and mapped json paths.

### Small cheatsheet

| Country | `lang` | `loc`  |
| ------- | ------ | ------ |
| US      | `en`   | `2840` |
| Germany | `de`   | `2276` |
| Spain   | `es`   | `2724` |
| Italy   | `it`   | `2380` |
| France  | `fr`   | `2250` |

```python
>>> from cobrseo.api.dataforseo_organic import save_dataforseo_organic_serps

>>> keywords = ['Industroyer', 'blackcat', 'revil', 'Moncler', 'Conti ransomware']
>>> destination_path = './serps'
>>> rewrite_serp = False
>>> token = 'API_KEY'

>>> save_dataforseo_organic_serps(
            keywords,
            destination_path,
            rewrite_serp,
            token
        )
[{'keyword': 'revil', 'path': './serps/d1c0dd7a20099294bfe3dba2c0b4e507.json'},
{'keyword': 'blackcat', 'path': './serps/5c55d71b4c47d141072cf0540c046d07.json'},
{'keyword': 'Industroyer', 'path': './serps/492ed356c6aa5e4bd9de4a81b4fa2add.json'},
{'keyword': 'Conti ransomware', 'path': './serps/6ba632a49d9e504bad1fde6f9281a2db.json'},
{'keyword': 'Moncler', 'path': './serps/faf6dd008e4b7640583c95e1cbbf1533.json'}]
```

### Version changes

##### `v0.0.1`

- Initial release.

##### `v0.0.2`

- Python version changed from 3.8 to 3.6 in PyPI.

##### `v0.0.3`

- `cobrseo.api.dataforseo_organic`:
  - Removed `dirs_to_check` parameter from `save_dataforseo_organic_serps` method.
    Flag `rewrite_serp` was added instead.
    Now SERP checking is performed in `destination_path` directory only.
  - Fixed bug with `save_dataforseo_organic_serps` return value.
    Now list with all keywords is returned, even with the ones that have already existed.
- Documentation updated.

##### `v0.0.4`

- `cobrseo.json.serp_processing`:

  - Added `get_knowledge_graph` method. If knowledge graph exists then `str` will be returned, otherwise - `None`.
  - Added `get_featured_snippet` method. Returns snippet in `dict`.

- `cobrseo.api.dataforseo_organic`:

  - Fixed bug with `null` in json. From now on, if GET request returns `None` in `result` section, it will continue sending GET request with same id untill it returns correct result.
  - Updated logging messages.

- `tests`:

  - New correct approuch for testing.
  - New tests for new methods.
  - Refactoring `source.py`.

- README.md available on pypi.org.
