I’ve been working on a project recently that requires some processing of video to add an overlay. The code was simple – using a library to do the real work, my C# code just organises the various contents and let it do its work. The more interesting part was that there are a lot of these needed, each with variations, so of course that’s where a database comes in. The video maker code looks for the first record that needs to be done, makes the video, marks the record as done, and moves on to the next one. Except actually the video library isn’t reliable, so it just exits, and a script looks at the exit code, and if not finished, it runs it again in a loop. Standard stuff.
So I had the code that was happy working on my dev machine, or on a Linux box I have set up for the purpose.
And away it goes making them one at a time. Thing is, each takes at least ten seconds, and doing tens of thousands therefore takes time, so I set up some more machines. But how to coordinate them? Well, the database is a SQLite file, so put that on a file server, point all the machines at it, and all is well. Except it isn’t because it was suffering locks, and after a while the whole file completely disappeared. Not sure how that happened, but I needed something better.
So I knocked up a little web service that is now in charge of the database. It has a single API: GetNextRecord which returns a record for a video maker to run. The video maker just uses the information in the record supplied, and puts the new video on the file server. It then exists, and gets re-run by the shell script, at which point it asks the web service for the next, and so on. It never updates the database itself.
The web service that is controlling the process is basically simple. First, it has a task that scans the file server every four minutes to see if the required files exist or not, updating the database as needed. It then has an in memory cache of records that need to be done, topped up automatically with records marked as needed. When a video maker asks for a task, the cached record is marked as active for ten minutes, before being deleted from the cache. All easy with the .Net tooling that C# gives – LINQ in particular. This means that the video maker has ten minutes to make the video and the scanning task will spot it now exists. But if the video maker failed, the file won’t be created, and will be scanned, and eventually end up back in the cache of video making tasks to be done.
My original thinking was that I’d then have to improve this to make it faster or something. But this simple two part controller works very well. And the code change from the original working video maker was simple – it used to access the database to get the next record, now a command line tells it to get the record from a web service.
The scaling up has worked really well – I have eleven machines running, and so far they’ve created over 100,000 videos. Sweet.
With the security updates for the OpenSSL project that come out, I started to face an issue as a developer who uses them in projects that I develop. I get an update, and I need to make sure that the new DLLs are put into the “distribution” directories for the projects, so that next time they are built the new DLL will be included “automatically”. But my projects are all over the place, and manually finding and remembering which ones to update is time consuming. Sure, I can use a search tool, but that leave the “go to the location and paste the files”.
So I wrote a tool to scan my disks and tell me which version is in situ. And then, if I want, to update them. The optional part is important because not all applications should be updated – there may be incompatibilities. An update is now much easier. You can put the new files, “ssleay32.dll” and “libeay32.dll” in the same directory as the updater if you want. It will create a “fileoptions.xml” file in the same directory too (so don’t put the updater in Program Files or other similar reduced access rights directory). The folders to scan can be separated with a semi-colon, so “C:\;D:\” will scan the C and D drives. Note that there must be a trailing ‘\’. Then scan (it is a background thread for speed). Click to select any you wish to update, and it will do so.
Supplied as-is.
One thing I often ponder is why I like the RemObjects SDK for my developments, when things like REST are “free” and perhaps more standard. The simple answer is that my code looks just like any other code I write. For this example, I have written a demonstration “chat” application. You log in, you can post messages, and you get all the messages from the server. This is the procedure that sends a message to the server:
procedure TfrmLogin.SendMessage(szMessage : String); begin m_xChatService.SendMessage(szMessage, SendMessageComplete, OnLinkError); end;
There’s no building up a URL, no encoding of parameters. Just a normal function call to a class member. If this wasn’t a WebBuilder function, I’d not have the SendMessageComplete callback – it would just wait until the call returned, but the browser is always asynchronous, so we have the callback:
procedure TfrmLogin.SendMessageComplete(nResult : Integer; szHumanResult: String);
begin
Report('Sent: ' + szHumanResult);
end;
That’s all there is in the client for normal coding. On the server, this is the code that implements the service:
function TChatService.SendMessage(const szMessageJSON: Utf8String; out szHumanResult: Utf8String): Integer; begin Result := 1; g_xMessageList.Add(szMessageJSON); end;
I have found Elevate Software’s WebBuilder product to be a very productive environment for me to use to develop web based user interfaces. As a long term Delphi developer, it suits me very well, allowing me to take all the language skills in Delphi, and create very capable Javascript single-page applications. It contains a core set of UI components which at first may seem to be lacking advanced options, but like Lego, you can make very nice complex components with the raw tools provide. I have implemented both an advanced grid, and a slider control using the provided TPanels, TLabels and other components.
The finished result:
I was in discussion with someone recently about how I go about doing my development, which uses a Javascript user interface built using the Elevate Software WebBuilder product, and a Windows service that is written using Delphi XE2. I was asked why I used the RemObjects SDK rather than any of the other mechanisms available. I have used it for many years and it has served me well, so it took a little time to work out what the benefit is – why do I pay money when there are other options “out of the box”?