youtube-dl, Python video download tool, on front page of Hacker News

By Vasudev Ram

youtube-dl is a video download tool written in Python.

I had blogged about youtube-dl a while ago, here:

youtube-dl, a YouTube downloader in Python [1]

and again some days later, here:

How to download PyCon US 2013 videos for offline viewing using youtube-dl

(The comments on the above post give some better / easier ways to download the videos than I gave in the post.)

Today I saw that a Hacker News thread about youtube-dl was on the front page of Hacker News for at least part of the day (up to the time of writing this). The thread is here:

youtube-dl (on Hacker News)

I scanned the thread and saw many comments saying that the tool is good, what different users are using it for, many advanced tips on how to use it, etc. The original creator of youtube-dl, Ricardo Garcia, as well as a top contributor and the current maintainer (Filippo Valsorda and Philipp Hagemeister, respectively) also participated in the HN thread, as HN users rg3, FiloSottile and phihag_, respectively. I got to know from the thread that youtube-dl has many contributors, and that its source code is updated quite frequently (for changes in video site formats and other issues), both points which I did not know earlier. (I did know that you can use it to update itself, using the -U option).

Overall, the HN thread is a worthwhile read, IMO, for people interested in downloading videos for offline viewing. The thread had over 130 comments at the time of writing this post.

(On a personal note, since I first got to know about youtube-dl and downloaded it, I've been using it a fair amount to download videos every now and then, for offline viewing, and it has worked pretty well. There were only a few times when it gave an error saying the video could not be downloaded, and I am not sure whether it was due to a problem with the tool, or with the video site.)

[1] My first post about youtube-dl also had a brief overview of its code, which may be of interest to some Pythonistas.

This other post which mentions youtube-dl may also be of interest:

The most-watched Python repositories on Github

since youtube-dl was one of those most-watched repositories, at the time of writing that post.

- Vasudev Ram - Dancing Bison Enterprises

Signup for emails about new products from me.

Contact me for Python consulting and training.

Share |

pafy - Python API for YouTube

By Vasudev Ram

PAFY (Python API For YouTube) is a Python library that does what the name says on the tin - it allows you to access YouTube videos programmatically, get information about them, download them, etc.

Full documentation for Pafy is here.

You can install Pafy with the command: pip install pafy

I tried out Pafy a little, and whatever I tried worked the way the docs said it would.

Here is a simple example of using Pafy to get info about a YouTube video and download it to your machine. I used the video 'Concurrency is not Parallelism' - by Rob Pike, co-inventor of the Go language, in the example. The video is also embedded below:

import pafy

# Concurrency is not parallelism - video URL = "https://www.youtube.com/watch?v=cN_DpYBzKso"
cinp_url = "https://www.youtube.com/watch?v=cN_DpYBzKso"

cinp_video = pafy.new(cinp_url)
cv = cinp_video

print "Info for video 'Concurrency is not Parallelism' by Rob Pike, Commander, Google:"
print 'Title:', cv.title
print 'Length:', cv.length
print 'Duration:', cv.duration
print 'Description:', cv.description

"""
This fragment prints info about the streams available in the video.
streams = cv.streams
for s in streams:
    print(s)
"""

# Get the best stream for the video.
best = cv.getbest()

# Download the best stream.
best.download(quiet=False, filepath=best.title + "." + best.extension)

The Pafy download() method not only downloads the video, it also shows the size downloaded so far as a percentage, the download speed, and the ETA for the download, in real time.

Also check out my earlier post on another YouTube tool in Python:

youtube-dl, a YouTube downloader in Python


- Vasudev Ram - Dancing Bison Enterprises - Python consulting and training

Contact Page

Share |

Micawber gets rich content from URLs

By Vasudev Ram



micawber is a Python library that enables you to "fetch rich content from URLs" of videos, music, etc. I guess micawber is named after the character Micawber in the novel "David Copperfield" by Charles Dickens.

coleifer/micawber on GitHub

Read the micawber docs

I wrote a small test program to try out micawber, by modifying one of the basic examples of its use. It fetches some of the data about 3 YouTube videos: Andres Segovia playing 'Asturias', Guido van Rossum's talk - 'In Python We Trust', and Luisa Fernandez singing 'Granada'.

Here is the program:

# test_micawber.py

import micawber

# load up rules for some default providers, such as youtube and flickr
providers = micawber.bootstrap_basic()

# Fetch the data for 3 video URLs
for url in [ 'http://www.youtube.com/watch?v=9efHwnFAkuA',
    'http://www.youtube.com/watch?v=mWB3oh1GPdo',
    'http://www.youtube.com/watch?v=UYGc9joeAD0' ]:
    
    retval = providers.request(url)
    for key in ['type', 'title', 'url', 'author_name', 
    'author_url', 'provider_name', 'provider_url', 'height',
    'width', 'thumbnail_url']:
        print key, ":", retval[key]

    print

# EOF

You can run it like this:

python test_micawber.py

And here is its output:

type : video
title : Andres Segovia - Asturias
url : http://www.youtube.com/watch?v=9efHwnFAkuA
author_name : ByTheSPiRiTs
author_url : http://www.youtube.com/user/ByTheSPiRiTs
provider_name : YouTube
provider_url : http://www.youtube.com/
height : 344
width : 459
thumbnail_url : http://i1.ytimg.com/vi/9efHwnFAkuA/hqdefault.jpg

type : video
title : In Python We Trust
url : http://www.youtube.com/watch?v=mWB3oh1GPdo
author_name : OreillyMedia
author_url : http://www.youtube.com/user/OreillyMedia
provider_name : YouTube
provider_url : http://www.youtube.com/
height : 270
width : 480
thumbnail_url : http://i1.ytimg.com/vi/mWB3oh1GPdo/hqdefault.jpg

type : video
title : Luisa Fernandez -  Granada
url : http://www.youtube.com/watch?v=UYGc9joeAD0
author_name : Dany Massart
author_url : http://www.youtube.com/user/DJCOOLDANY
provider_name : YouTube
provider_url : http://www.youtube.com/
height : 344
width : 459
thumbnail_url : http://i1.ytimg.com/vi/UYGc9joeAD0/hqdefault.jpg

- Vasudev Ram - Dancing Bison Enterprises

Contact me

Share |

How to download PyCon US 2013 videos for offline viewing using youtube-dl

By Vasudev Ram


As Pythonistas would know, PyCon US 2013 happened recently.

While reading Python news on the Net, I saw that the videos of the conference were being uploaded to pyvideo.org.

I browsed the list of videos there. Since I had recently come across youtube-dl, a YouTube downloader tool, I thought of trying to download the PyCon videos using it.

When I went to the pyvideo site to do that, and played one or two videos available there, I was not sure at first whether they were from YouYube or not, so was not sure whether I could download them using youtube-dl.

A little experimentation showed that they were in fact hosted on YouTube, and I was able to find out how to download them, and then downloaded a few of them.

So I'm giving the steps for doing that below:

(Note: these steps worked for me - I could download a few videos using the steps, and none have given an error so far, but YMMV, depending on your OS, browser, Flash Player version, or other factors.)

Update: before (or after) reading these steps, you may wish to read the comments on this post. Some readers commented, and gave simpler methods of doing the same thing, in particular, some said that there is a direct link to download the video on some or all of the pyvideo video pages, and some said that the URL-fixing step is not needed since you can just right-click on the video to get the right URL without the extraneous stuff. Also, renaming the video to remove the extraneous stuff in the name can be avoided by using some command-line options of youtube-dl.

- Go to pyvideo.org in your browser.

- Select the category called PyCon US 2013 videos.

- Select one of the videos from that category and open the link to the video in another browser tab (or in the same tab).

For example, try Guido van Rossum's keynote at PyCon US 2013.

- Click the share video icon at the top right of the video screen; it looks like a V (with small knobs at its vertices), but rotated to the right by 90 degrees.

- The top left of the video screen will now show the URL for the video (along with buttons to share it on Facebook, Twitter, etc.)

- Hover your mouse over that URL and then right click it. You may have to right-click twice, or first click on the URL and then right-click on it.

- From the Adobe Flash Player pop-up menu that appears, select "Copy video URL".

- Now, open a DOS command prompt. I am using Windows in this example; adjust the steps for Linux or Mac OS X, i.e. use a bash or other shell and modify the steps for copy and paste below, to suit your OS.

- At the prompt (Windows), type the partial command:

youtube-dl -t (followed by a space)

- Then type Alt-Space, then E, then P.

- That brings up the Windows system menu for the DOS window, and then selects the Edit menu in it, and then the Paste option from that menu.

- So the video URL will get pasted to the end of youtube-dl command you typed earlier.

- Now, if you just try to run the command as it is, it may not work (for downloading by youtube-dl), because of some extra parameters in the URL, that youtube-dl cannot handle.

- For example, here is the copied video URL for the GvR keynote:

http://www.youtube.com/watch?feature=player_embedded&v=n6dUWXRdt48

- Edit the URL using arrow keys, backspace, delete key, etc., to remove the part from "feature" upto "embedded&"; in other words (using HTTP request terms), remove all the key-value pairs except for the "v=something" pair.

- Using the above GvR keynote example, the command-line should now look like this:

youtube-dl -t http://www.youtube.com/watch?v=n6dUWXRdt48

- Now hit Enter, and wait while the video downloads. The -t option to youtube-dl will add the title of the video as a prefix to the download filename. I typically rename the file to remove the cryptic characters after the title, like this:

c:\downloads> ren Keynote-n6dUWXRdt48.flv GvR-Keynote.flv

- Now you can play the video in VLC, MPlayer or any other suitable player.

Enjoy.

- Vasudev Ram - Dancing Bison Enterprises

Share |

youtube-dl, a YouTube downloader in Python

I came across youtube-dl recently. As the title says, it is a YouTube downloader tool, written in Python.

(Update: I made a typo in the title of this post - wrote YouRube. Unintended :-), and fixed now. The post URL can't be fixed, though, except by deleting and re-entering the post.)

I downloaded the tool (it comes both as a binary and as source) and ran the tool on Windows, at the command line, to download a favorite video of mine from YouTube. It worked fine.

youtube-dl has various command-line options that you can set to configure its behavior.

Here is the documentation for youtube-dl.

Then I took a look at the source code. The code for the tool is in the public domain.

It is a few thousand lines long, and divided into 5 or 6 Python source files. There is one file called FileDownloader, another called InfoExtractor, and a few other supporting files.

I've just had a short look at the code so far, but could figure out a few things about the design, and am giving an overview of that below. Going to check out the code in more detail later. It is a non-trivial app and can serve as a good codebase to study.

Instances of the FileDownloader class and InfoExtractor classes register themselves with each other. Actually, a number of InfoExtractors can be registered with a FileDownloader instance. The FileDownloader then passes the input video URL (which the user wants to download) to the chain of InfoExtractors, and uses the first one which reports that it can handle the request, to extract some required info about the video. The FileDownloader itself then does the actual downloading of the file (if requested by the user; it can do other things than download the file, too). Some of its features are:

- Supports multiple YouTube formats - they link to this list of YouTube formats on Wikipedia, so I guess they must support at least some of them; I haven't tried out formats other than the default, but I may later. Some of those YouTube formats include HD video (higher resolution) and 3GP (for mobile). If youtube-dl supports mobile, it could be used to download videos for mobile to your PC, and then transfer it to your mobile, for offline viewing, if you can find an offline video player app for your mobile. Going to try that out for my Android phone.

- Supports multiple video sites, not just YouTube. Some of them are vimeo, Google Video (not sure if that is still operational), blip.tv, SoundCloud, and infoq.

- The tool youtube-dl can be used to update itself, and they recommend you do it often, with the command:

youtube-dl -U

Since YouTube videos are usually in Flash video format, you can play them on your Windows or Linux PC using VLC or MPlayer.

The video I downloaded was in MP4 format, and I was able to play it in Windows Media Player (didn't have VLC just then on that machine), and it did play, though the audio was low and the picture quailty was not that clear, as compared to playing the same video in YouTube via the web client. Windows Media Player gave a message that the MP4 format was unknown, but was able to play it anyway, though with the stated limitations.

youtube-dl can use either urllib or urllib2 from the Python standard library, can do download rate-limiting if asked to, and many other things, as part of its features.

An interesting Python tool, overall.

On a related note, a lot of YouTube itself is written in Python. I had seen a video about that by one of the YouTube engineers, some time ago.

Here's the summary of a talk at PyCon 2012 on Scaling YouTube - it mentions that most of YouTube is written in Python, and that YouTube is one of the largest web sites in the world, that uses Python. I just saw on Wikipedia in the entry about YouTube that it may be the third-most viewed site in the world.

vitess - scalable RPC interface to MySQL, used by YouTube

vitess - Scaling MySQL databases for the web - Google Project Hosting

Vitess, used at YouTube, is a scalable RPC interface to MySQL, with sharding, limited ACID support, Python and Go client interfaces, and other features.

.

Python used for Dropbox's Linux, Windows and Mac client software

By Vasudev Ram


In an O'Reilly Radar interview, Python creator Guido van Rossum (GvR) says that the Linux, Windows and Mac client software for Dropbox ( the widely used cloud file-storage and sharing service), are all implemented in Python.

The O'Reilly article also mentions GvR's Google+ post on improving Python performance. I had heard about that post recently, but when I tried to view it, I kept getting an error "This post could not be found".

- Vasudev Ram - Dancing Bison Enterprises

Share |