You’re ready to audit and optimize your (or your clients’) YouTube channel or you want to see what a competitor is doing with their YouTube videos. But you run into the problem that YouTube makes it kind of tough to scrape and export video information.
You can’t really crawl YouTube like you can a website – it’s too large and there’s no way to control your crawl. I ran into this problem a few weeks ago while trying to map out videos and optimize titles & tags in bulk.
Here’s how to scrape & export video information from YouTube without buying sketchy blackhat scraper software.
1. Get Your Setup Ready
You’re going to need:
Scraper for Chrome (free) to scrape the video URLs.
Google Sheets (free) to organize the data.
Screaming Frog (free up to 500 URLs) to crawl your videos.
2. Load up all your YouTube videos
You’re going to just keep loading more videos until you can’t load anymore.
*Some readers are reporting that YouTube will occasionally block scraping on their main app. To get around that, before pressing Scrape Similar, change the URL to m.youtube.com. Then you’ll be redirected to a page which ends in =desktop. Same content, but no scraper blocking Javascript (you can also hunt down & block the blocking script as well). h/t Exequiel Plaza.
3. Scrape the YouTube videos
Right-click on any video link, then click Scrape Similar.
Double-check that it has scraped all the videos, then Export to Google Docs.
4. Clean up your URLs and save to text file
In your Google Doc, add https://www.youtube.com onto all the URLs
Then (if necessary) strip any unneeded parameters from the videos (such as &list)
5. Get your list into Screaming Frog
Copy and paste your list of URLs into a TextEdit (Mac) or Notepad (Windows) and Save as a .txt file
Now open Screaming Frog and set it to List Mode, then crawl.
6. Get it back into a spreadsheet
Export your Screaming Frog crawl to a .csv or .xlsx and move the data to either Microsoft Excel or back to Google Sheets.
7. Use the data!
You will be primarily be looking at the Title and Meta Keywords columns. Many SEOs know that YouTube uses your video tags as a relevance & quality signal.
However, many SEOs do not know that YouTube stores the video Tag information in the otherwise useless Meta Keywords field.
Aside – just so there’s no confusion, Google does not use the meta keywords tag to gauge relevance of websites. YouTube does use Tags to judge relevance of videos hosted on YouTube. The tags are stored in the meta keywords field.
You can also use the data to judge meta descriptions & word count at scale to spot optimization opportunities.
Bonus!
Screaming Frog also has a Custom Extraction function. If you can find the Xpath of any HTML element on a YouTube page, then you can scrape it.
For example, if you want a point in time of YouTube views on your list of videos. Just grab the Xpath of the element via right-click, Inspect Element, Copy Xpath.
And then drop that Xpath in the Custom Extraction section of Screaming Frog.
Note – you may have to fiddle with the User Agent to keep from being blocked. Either way – it’s a useful tool for quick analysis.
2nd Note – you may also have to change the Rendering setting under Spider to “Javascript” instead of “Text Only” since YouTube loads a lot of page assets dynamically (notably, the view count).
Next Steps & Additional Thoughts
Go grab Scraper for Chrome and Screaming Frog and start pulling your YouTube information into a spreadsheet! Once you have your data in order, be sure to take advantage of YouTube’s bulk editor to quickly edit tags & meta information.
Don’t forget that the Meta Keywords field is on every YouTube video, so definitely spy on and leverage your competitor’s YouTube video information.
Read more about analyzing your own YouTube Analytics here.
