Metadata-Version: 2.1
Name: videolab-youtube-crawler
Version: 1.2.1
Summary: Clark University, Package for YouTube crawler and cleaning data
Home-page: https://github.com/shuoniu89/VIDEOLab-youtube-analysis
Author: Shuo Niu
Author-email: ShNiu@clarku.edu
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: configparser
Requires-Dist: datetime
Requires-Dist: pytz
Requires-Dist: pandas
Requires-Dist: isodate
Requires-Dist: xlrd
Requires-Dist: youtube-transcript-api
Requires-Dist: google-api-python-client

# videolab-youtube-crawler

This is an integrated tool for crawling YouTube data and decode JSON. See the use cases for examples.

This tool is developed by Shuo Niu at Clark University. 

Pypi page: "https://pypi.org/project/videolab-youtube-crawler/"



## Installing
To install,

``pip install videolab-youtube-crawler``

The crawler needs multiple other packages to function:

You may need to install those packages:

``pip install youtube-transcript-api``

``pip install google-api-python-client``

``pip install yt-dlp``

``pip install deepmultilingualpunctuation``

Also install the packages if you don't have them in your vent

``pip install pytz``

``pip install pandas``

``pip install asyncio``

``pip install isodate``

``pip install configparser``

``pip install httplib2``


## Upgrading
To upgrade

``pip install videolab-youtube-crawler --upgrade``

Go to the project folder, delete config.ini if it is already there.


## YouTube API Key
- Go to https://cloud.google.com/, click console, and create a project. Under Credentials, copy the API key.
- In your project folder, create a "DEVELOPER_KEY.txt" file (must be this file name) and paste your API key. 
- You can use multiple API keys by putting them on different lines in DEVELOPER_KEY.txt. 
- The crawler will use up all quotas of one key and try next one, until all quotas are used up.



## Example usage
Case 1: crawl videos by keywords, 
```
import videolab_youtube_crawler as ytcrawler

# create folders for search videos
ytcrawler.init_config()
searcher = ytcrawler.VideoSearcher()
searcher.search("Generative AI",
                start_day=1,
                start_month=1,
                start_year=2023,
                end_day=31,
                end_month=12,
                end_year=2023,
                day_span=30)
searcher.search("Artificial Intelligence",
                start_day=1,
                start_month=1,
                start_year=2023,
                end_day=31,
                end_month=12,
                end_year=2023,
                day_span=30)
                
searcher.merge_to_workfile(destination="DATA/list_video.csv")

# Get all video and channel information
video_crawler = ytcrawler.VideoCrawler()
video_crawler.crawl_videos_in_list(video_list_workfile=f"DATA/list_video.csv")
video_crawler.json_to_csv()

# Get all channel information from the list generated by merge_to_workfile()
channel_crawler = ytcrawler.ChannelCrawler()
channel_crawler.crawl_channel_of_videos(video_list_workfile=f"DATA/list_video.csv")
channel_crawler.json_to_csv()

# Get all comments from the list generated by merge_to_workfile()
comment_crawler = ytcrawler.CommentCrawler()
comment_crawler.crawl_comments_of_videos(video_list_workfile=f"DATA/list_video.csv", comment_page=2)
comment_crawler.json_to_csv()

# Get all subtitles from the list generated by merge_to_workfile()
# Since the default YouTube cpations may not contain punctuations, use the split_caption_by_sentences to add punctuations and split 
# the captions by sentences. It will automatically align the timestamps.
subtitle_crawler = ytcrawler.SubtitleCrawler()
subtitle_crawler.crawl_subtitles_in_list()
subtitle_crawler.split_caption_by_sentences()
```

Case 2: Search a list of channels by search keys, then crawl all videos belonging to those channels.
```
import videolab_youtube_crawler as ytcrawler

# Create folders for search videos
ytcrawler.init_config()
searcher = ytcrawler.ChannelSearcher()
searcher.search("ADHD")
searcher.search("PTSD")
searcher.merge_to_workfile(destination="DATA/channel_list.csv")

# Get channel metadata in a given channel list
channel_crawller = ytcrawler.ChannelCrawler()
channel_crawller.crawl_channel_in_list(channel_list_workfile="DATA/channel_list.csv")
channel_crawller.json_to_csv()

# This step usually has too many videos. Filter the channel before crawling.
channel_crawller.crawl_all_videos_of_channels(channel_list_workfile='DATA/channel_list.csv')

# Merge the searched videos into a workfile
video_searcher = ytcrawler.VideoSearcher()
video_searcher.merge_to_workfile()

# Search all videos in the workfile
video_crawler = ytcrawler.VideoCrawler()
video_crawler.crawl_videos_in_list(video_list_workfile='DATA/list_video.csv')
video_crawler.json_to_csv()

# Get all comments from the list generated by merge_to_workfile()
comment_crawler = ytcrawler.CommentCrawler()
comment_crawler.crawl_comments_of_videos(video_list_workfile=f"DATA/list_video.csv", comment_page=2)
comment_crawler.json_to_csv()

# Get all subtitles from the list generated by merge_to_workfile()
subtitle_crawler = ytcrawler.SubtitleCrawler()
subtitle_crawler.crawl_subtitles_in_list()
subtitle_crawler.split_caption_by_sentences()
```

Case 3: You already have a list of channels. You want to crawl all videos of the channels in the list:
```
import pandas as pd

import videolab_youtube_crawler as ytcrawler

ytcrawler.init_config()

# Get channel metadata in a given channel list
channel_crawller = ytcrawler.ChannelCrawler()
channel_crawller.crawl_channel_in_list(channel_list_workfile="speech_disability_channels.csv")
channel_crawller.json_to_csv()

# This step usually has too many videos. Filter the channel before crawling.
channel_crawller.crawl_all_videos_of_channels(channel_list_workfile='speech_disability_channels.csv')

# Merge the searched videos into a workfile
video_searcher = ytcrawler.VideoSearcher()
video_searcher.merge_to_workfile()

# Search all videos in the workfile
video_crawler = ytcrawler.VideoCrawler()
video_crawler.crawl_videos_in_list(video_list_workfile='DATA/list_video.csv')
video_crawler.json_to_csv()

# Get all comments from the list generated by merge_to_workfile()
comment_crawler = ytcrawler.CommentCrawler()
comment_crawler.crawl_comments_of_videos(video_list_workfile=f"DATA/list_video.csv", comment_page=10)
comment_crawler.json_to_csv()

# Get all subtitles from the list generated by merge_to_workfile()
subtitle_crawler = ytcrawler.SubtitleCrawler()
subtitle_crawler.crawl_subtitles_in_list()
subtitle_crawler.split_caption_by_sentences()

# Download the video streams. to use better quality, change the quality='best'
video_downloader = ytcrawler.VideoDownloader()
video_downloader.download_videos_in_list(video_list_workfile="DATA/list_video.csv", quality='worst', audio=False,
                                         batch=200)
video_downloader.download_videos_in_list(video_list_workfile="DATA/list_video.csv", quality='worst', audio=True, video_data_dir='test')

```




