DotNet
I’ve been working on a project recently that requires some processing of video to add an overlay. The code was simple – using a library to do the real work, my C# code just organises the various contents and let it do its work. The more interesting part was that there are a lot of these needed, each with variations, so of course that’s where a database comes in. The video maker code looks for the first record that needs to be done, makes the video, marks the record as done, and moves on to the next one. Except actually the video library isn’t reliable, so it just exits, and a script looks at the exit code, and if not finished, it runs it again in a loop. Standard stuff.
So I had the code that was happy working on my dev machine, or on a Linux box I have set up for the purpose.
And away it goes making them one at a time. Thing is, each takes at least ten seconds, and doing tens of thousands therefore takes time, so I set up some more machines. But how to coordinate them? Well, the database is a SQLite file, so put that on a file server, point all the machines at it, and all is well. Except it isn’t because it was suffering locks, and after a while the whole file completely disappeared. Not sure how that happened, but I needed something better.
So I knocked up a little web service that is now in charge of the database. It has a single API: GetNextRecord which returns a record for a video maker to run. The video maker just uses the information in the record supplied, and puts the new video on the file server. It then exists, and gets re-run by the shell script, at which point it asks the web service for the next, and so on. It never updates the database itself.
The web service that is controlling the process is basically simple. First, it has a task that scans the file server every four minutes to see if the required files exist or not, updating the database as needed. It then has an in memory cache of records that need to be done, topped up automatically with records marked as needed. When a video maker asks for a task, the cached record is marked as active for ten minutes, before being deleted from the cache. All easy with the .Net tooling that C# gives – LINQ in particular. This means that the video maker has ten minutes to make the video and the scanning task will spot it now exists. But if the video maker failed, the file won’t be created, and will be scanned, and eventually end up back in the cache of video making tasks to be done.
My original thinking was that I’d then have to improve this to make it faster or something. But this simple two part controller works very well. And the code change from the original working video maker was simple – it used to access the database to get the next record, now a command line tells it to get the record from a web service.
The scaling up has worked really well – I have eleven machines running, and so far they’ve created over 100,000 videos. Sweet.