English

Building a home server

A few years ago, I started looking into streamlining our movie-watching experience. Downloading, organizing and browsing through our movie collection became increasingly difficult as it grew.

Our original setup was fairly basic: I would torrent movies on my laptop, copy them on a USB stick and pray that the television's media player could read the video format. When it didn't, we had to forego the luxury of a remote and connect the laptop directly to the television.

Once I purchased my first home server, I installed a light version of Linux with XBMC, hoping that the USB drive would become a thing of the past. Moreover, XBMC would automatically rename files and fetch movie covers, ratings and descriptions from IMDB and we wouldn't have to.

XBMC is very interesting ...on paper

XBMC is very interesting ...on paper

Despite my best efforts, XBMC failed to deliver on its promises. Files were matched to the wrong movies all the time, and the interface was awkward to use, even with a wireless mouse and keyboard. I have also toyed with CouchPotato, but it was equally frustrating, as it often downloaded torrents in the wrong language.

After less than a month, we bypassed all of these tools and simply launched videos from the file browser. This new workflow was functional and reliable, but it became increasingly hard to find what to watch among a disorganized collection of torrented files.

Renaming the files was not possible until the torrents were finished seeding. Once in a while, I would manually create hard links between the torrented files and a directory where all videos were appropriately named. It was tedious manual work, and it meant freshly downloaded movies would take a while to show up in the "clean" movie folder.

Eventually, I gave up wrangling with less-than-ideal solutions and began building my own. My goal was to make finding, downloading, renaming, listing and playing the movies as simple as possible.

Seamless torrenting

Since my home server is directly connected to my television, I do not need to copy movies to USB drives to play them anymore. I use Transmission on my home server to download torrents, and I access it through its web interface. It's simple, reliable and flexible.

The Transmission web interface, integrated to the home server

I installed Torrent to Transmission on my laptop so I could start torrents on my server in two clicks.

Torrent to Transmission lets you start torrents on a remote machine

Instead of counting on unreliable heuristics to pair videos to the correct movies, I do it manually from that interface. Once a torrent is finished downloading, the video files it contains are listed, and a box lets you enter the movie's title.

The OMDb API is used to suggest movie titles. The interface can be operated without using the mouse, so a dozen files can be matched the correct movies in a few seconds.

Once the match is made, a hard link is then created in a separate directory using the movie title and the year. Inglourious Basteds (2009).mp4 in the movies folder points to IngloriousBasterd.BRRIP.2014.Team_NRG.mp4 in the downloads folder. With a hard link, one file is at two different places, but does not take double the space. More importantly, when the torrent is finished seeding, I can just delete the original file and the renamed one will stay.

The messy downloads folder

Screen Shot 2016-01-03 at 2.47.34 PM

The neatly organized movies folder (.srt files are subtitles)

Subtitles are also copied and renamed. I needed them to follow the movies, as I like to watch them in their original languages.

I eventually added the support for multi-part movies, as I also had miniseries in my collection. This allows me to regroup them in the interface, but to play them separately.

Screen Shot 2016-01-02 at 5.53.35 PM

Being able to organize my movie collection from any computer using a single, clear interface was a great step forward. There is no need to manage downloads or mess with files anymore.

Creating a great experience

With the file management situation sorted out, the next objective was to make it a breeze to pick a movie and play it on the television.

The list view shows all movies sorted by title, release date or IMDB rating. Clicking a movie shows its cover and description in the sidebar. This is perfect if you already know what you want to watch or you need to manage the movie collection. However, it's not a very good way to pick something to watch.

The list view

The cover view, on the other hand, is much better suited for those nights when you don't have anything in mind. The most recent additions are shown first.

The cover view

Handling miniseries gracefully was a bit more complicated. I grouped them under a same cover, and the miniseries are only marked as watched once all parts are watched. The Django ORM wouldn't let me have it, so I had to write my own queries to get them sorted this way.

The movies you mark as watched are sent to the bottom of the cover list so you always get a fresh selection of titles. In the list, they show as slightly greyed out.

Cinema anywhere

DSC01807

At this point, the media center did exactly what I needed to do, and much better so than XBMC. However, I often wanted to watch movies from my tablet in bed, or on my laptop "between" classes. In that regard, Netflix was far more convenient, despite its limited selection.

In order to stream my movies on the web, I needed to convert them all to H.264, a format that is supported by pretty much any video player. Since my upload speed at home is limited, I also needed to reduce each video's size so it could stream faster than it plays (about 1.5mbps with my current connection).

Since making videos small enough to stream smoothly meant a loss in quality, I opted to keep the original HD videos too, which meant each movie is stored twice. The decreasing price of storage made this a triviality.

However, the CPU power needed to convert several hundred movies was far from trivial. My original Atom-powered home server struggled to convert four movies a day. Luckily, switching laptops freed up a reasonably powerful i5-powered ThinkPad to replace my dated Acer Revo as a home server. I was now able to convert about 12 movies a day without overtaxing the hardware.

Using the following ffmpeg command, I converted all movies to H.264.

# Converts a movie for web playback using ffmpeg
/usr/bin/ffmpeg -i "$1" \
-codec:v libx264 \
-profile:v high \
-preset slow \
-movflags faststart \
-b:v 800k \
-maxrate 1000k \
-b:a 128k \
-bufsize 1000k \
-vf scale="trunc(oh*a/2)*2:480" \
-threads 0 \
-ac 2 \
-async 1 \
-loglevel warning \
-codec:a libvo_aacenc "$2" 2> ${LOG_FILE}

The quality is good enough for anything but a large television, for which the HD version is still there. I ran into audio sync issues, but after a bit of research, it seems that only the Chrome player shows a delay between the audio and the video. VLC plays the videos correctly.

Since HTML 5 videos support subtitles, I automatically convert SRT subtitles to WEBVTT with pycaption when the video is finished converting and load them with the video. This works fine for torrents that include .srt subtitles, but many videos have built-in subtitles. I have implemented a solution that extracts them using enzyme and mkvextract, but it still needs some testing.

A video with matching subtitles in Chrome's native player

A video with matching subtitles in Chrome on Android

Movies in H.264 have the benefit of being playable on pretty much any device, so I can simply save the movies to my phone before a long flight.

Refining the experience

A few interface tweaks greatly improved the movie-watching experience. In addition to many style tweaks, I have fixed some minor frustrations with the interface.

When movies are played on the web, I use JavaScript to extract the video position. This allows me to resume the movie at the same place later. This tiny addition made it much more enjoyable to finish a movie in multiple sittings.

//Submits the current video position to the server every 10 seconds
$(document).ready(function(){
    var duration = 1000000;

    $('video').bind('canplay', function() {
        var interval = setInterval(
            function(){
                var seconds = Math.floor($('video')[0].currentTime);

                //Reset the counter if within 10 minutes from the end
                if(seconds > video.duration - 600) seconds = 0;

                $.post(
                    "/movie/13/position/",
                    {'seconds': seconds}
                );
            },
            10*1000
        );
    });
});

When a movie is unfinished, it shows at the top of the list to force me to finish it. When I mark a movie as watched, its position is reset.

A dashboard for everything

At some point, it becomes hard to keep an eye on the important things. A script might stop working and go unnoticed for weeks, or I might run dangerously low on funds in my debit account. I wanted a dashboard that could tell me the state of things at a glance.

My home server is far more than just a media server; it is connected in a way or another to every aspect of my life. It hosts copies of almost all the data I hold, runs scripts that automate my finances, keeps an eye on all of my servers and even used to control appliances. It thus seemed like the ideal candidate to host this dashboard.

Before that, I wanted to achieve the same goals using an assistant not unlike the Amazon Echo that I called Winston. Unfortunately, it was impossible to get sufficiently reliable voice recognition, so I canned the project. A simple dashboard was simpler, more reliable and more efficient.

Screen-Shot-2016-01-17-at-12.34.36-AM

The dashboard page

The stacked chart is generated by the Google Graph API. Small scripts fetch the balance from my TD and Commerzbank accounts every day at 5PM. The TD script is on GitHub, if you are curious. Since I have bank accounts in Canada and in Germany, I use the Open Exchange Rates API to get a total balance in a single currency.

To update the dashboard, you only need to make a POST request to a small API. This makes it simple to update the dashboard from scripts in any language.

In case of failure, a notification is immediately dispatched to my phone using Pushover. If a website goes down, my backups stop working, my balance goes too low or my hard drive is almost full, I get notified before any damage is done. If I don't get any notifications, I know that everything is working as it should.

Security

Screen Shot 2016-01-17 at 1.33.22 AM

Although I wanted to share my movie collection with others, I did not to let anyone add torrents, manage my collection or glance at anything private. I also didn't want to have a friend remotely start a movie on my television in the middle of the night.

In order to fix these security risks, I have created two account levels: people with administrative privileges who can see and do everything and visitor accounts that can only play movies in the browser.

Future plans

My home server has been running without major changes for several months, so I consider it to be fairly stable at this point. I would like to improve error handling for failing movie conversions and perhaps open source the whole thing, but I consider the project to be done for the most part.

Laisser un commentaire