Developer
Developer related
I’ve been working on a project recently that requires some processing of video to add an overlay. The code was simple – using a library to do the real work, my C# code just organises the various contents and let it do its work. The more interesting part was that there are a lot of these needed, each with variations, so of course that’s where a database comes in. The video maker code looks for the first record that needs to be done, makes the video, marks the record as done, and moves on to the next one. Except actually the video library isn’t reliable, so it just exits, and a script looks at the exit code, and if not finished, it runs it again in a loop. Standard stuff.
So I had the code that was happy working on my dev machine, or on a Linux box I have set up for the purpose.
And away it goes making them one at a time. Thing is, each takes at least ten seconds, and doing tens of thousands therefore takes time, so I set up some more machines. But how to coordinate them? Well, the database is a SQLite file, so put that on a file server, point all the machines at it, and all is well. Except it isn’t because it was suffering locks, and after a while the whole file completely disappeared. Not sure how that happened, but I needed something better.
So I knocked up a little web service that is now in charge of the database. It has a single API: GetNextRecord which returns a record for a video maker to run. The video maker just uses the information in the record supplied, and puts the new video on the file server. It then exists, and gets re-run by the shell script, at which point it asks the web service for the next, and so on. It never updates the database itself.
The web service that is controlling the process is basically simple. First, it has a task that scans the file server every four minutes to see if the required files exist or not, updating the database as needed. It then has an in memory cache of records that need to be done, topped up automatically with records marked as needed. When a video maker asks for a task, the cached record is marked as active for ten minutes, before being deleted from the cache. All easy with the .Net tooling that C# gives – LINQ in particular. This means that the video maker has ten minutes to make the video and the scanning task will spot it now exists. But if the video maker failed, the file won’t be created, and will be scanned, and eventually end up back in the cache of video making tasks to be done.
My original thinking was that I’d then have to improve this to make it faster or something. But this simple two part controller works very well. And the code change from the original working video maker was simple – it used to access the database to get the next record, now a command line tells it to get the record from a web service.
The scaling up has worked really well – I have eleven machines running, and so far they’ve created over 100,000 videos. Sweet.
I’d just like to put together my thoughts on how well I think that Elevate WebBuilder (EWB) fits the requirements of many developers today, particularly Delphi developers. In the last year, I’ve done a lot of thinking about the tools I use, and what I need from them. What I think of course doesn’t apply to everyone, but it might help someone in their thinking.
The key for me was that mobile is here. A friend who moved from Delphi to .Net a long time ago had been telling me that the desktop is dead for most purposes, and given that I develop both my own products and for others, and I was doing desktop stuff, I knew it to be “wrong”, but it is also right of course – it is fading. You cannot develop for desktop alone. Apple’s iOS was for a long time the main target, with Android a second stage. Delphi recognised this too, and started targeting iOS. But while they floundered getting something sensible together, the world changed, and our customers are not just using iOS, and not only Android as a primary target too, but Windows phone and the web and anything you can think of. I was not happy with the way XE4 turned out, with language changes too. I like the single UI platform to target, and that wasn’t available in the “pascal” language options, nor C# offerings. EWB came along at the right time for me, and I have been amazed at the close control it gives. I can easily respond to the screen size of the device that is running it (particularly with the new option coming in 1.02 (and now released) that I’ve been hacking in recently myself). I’ve been listening to the Tablet Show podcast recently, and I am a fan of the “application theme” rather than “device style”, given that our application is going to be used across multiple devices. EWB does this nicely – making my own theme to style our application has been easy.
For some time now, I’ve struggled to work out how to build the WiX files that are generated by MSI Factory in FinalBuilder and include the bootstrapper. Building WiX files in FinalBuilder is dead easy – it has actions for the WiX Candle and the WiX linker so you can compile the files that MSI Factory generates (or any other WiX files of course) quite easily. But once you have your MSI, it is nice to have the EXE wrapper for bootstrapping it. Why does this matter? Because if you issue an update to your MSI file, and then double-click to run it, Windows will tell the user they must manually uninstall the app before they can install the update. Not so friendly.
Okay, so what’s so hard? The bootstrap compiler takes a number of options, and they include the GUIDs that are needed to check if it is installed already. The problem is that you don’t know the GUID in the build. It turns out that the answer is simple, but it took some time to find, so I’m blogging it so I remember for next time.
The MSI Factory software comes with a command line tool called “IRMakeBootstrap.exe”. This takes an XML configuration file and the name of the output file as parameters, and the product name, but also then needs three parameters that are key to proper operation.
The key ones are -var:PackageCode and -var:ProductCode. Now if you look at your WiX file you will see that it has a line like:
<Product Id="D03192D5-40A2-4EAC-B1A1-9BACE96AFE78" Name="My App Name" Version="1.0.0" Manufacturer="My Company" UpgradeCode="76D9BEB2-C077-4B7C-A91E-DF294F4C3457" Language="1033">
The “Id” matches the “Product Code” , and the “UpgradeCode” matches the Upgrade code, in the Project settings dialog of MSI Factory. However, the PackageCode is not in the file at all. There is a Package node in the XML, but no value. If you look in MSI Factories dialog, there is a “Package ID” on the Package tab, but it says “leave blank to auto-generate”. But if that is left blank, I can’t find out what it is to set it to something useful, and thus pass to the bootstrap maker.
Okay, so what is the solution? Simple. Define a GUID in the dialog for the Package ID in the MSI Factory dialog. This is normally a “bad thing”, because you want to to change each time (pretty much essential). But you are doing better anyway by using FinalBuilder (or other build tool?) and FinalBuilder is able to generate a GUID just fine. So, set it to a valid GUID string, and then generate the WiX files.
<Package Id="59D9FCDF-13BE-4AD7-841B-1F11D2E74500" ...
You will now find that the XML contains an “Id” for the package too. Haha! Now you have what you need to automate it. The value in the XML can now be replaced in your script, and also passed through to the bootstrap builder. Note that the GUIDs passed to the bootstrap builder need the curly brackets surrounding them. This ensures that they match properly.
Finally, you can check that it all works properly by running your install EXE with the “/Log:c:\install.txt” parameter which causes it to output a log of its activities.
The final parameters for the bootstrap maker is:
"D:\Src\bootstrap_config.xml" "D:\Src\MySetup.exe" -var:AllowExtractMSI=1 -var:ProductName="Product Name" -var:PackageCode="{%INST_PACKAGECODE%}" -var:ProductCode="{%INST_PRODUCTCODE%}" -var:ProductVersion=%VERSION_MAJOR%.%VERSION_MINOR%.%VERSION_BUILD%
Something that has bothered me for a while is the hassle in putting together all the pieces to sign my download files. I finally got around to looking it all up, and it isn’t easy as you have to piece all the things together. I hope that this will give you an insight into how it can really work, since I managed to actually achieve what I wanted and sign my code. This article is a list of the steps I had to take, and you will probably need to review the commands yourself if you have problems.
Update 2006:How things change. Since I wrote this original article, things in the codesigning world have become a lot easier. I’ve even renewed the two year certificate I bought – but watch out that you get the renewal from the expiry of the first one or you could lose many days! The main change though is that Microsoft made the signcode executable very hard to find, if not impossible. The good news is that there is a much better option available and, although it isn’t free, it is more comprehensive and offers a GUI mode for your initial testing and a command line option (signcode compatible) for your development script. The best improvement is that you can specify the password for the certificate, which means you don’t have to have the password dialog pop up all the time. Full details, and links to certificate suppliers and a step-by-step guide are available from the X2Net SignCode web page. I’ve left the original article below as it is still a good summary of the tools that might be useful to people, and how to transform things. |
The process
Get a certificate from http://www.ascertia.com/onlineCA/Issuer/CerIssue.aspx who will do a free code signing certificate. Obviously any alternative is good, but this will prove the concept for you, and you can go buy another from them or elsewhere later. The email address is included in the certificate, so use a sensible one you are happy for the world to see. Accept the installation of the certificate into the browser as it won’t be emailed even though they say it will. Make sure you chose exportable.
Then get the Microsoft code signing stuff (codesigningx86.exe) from the MSDN web site (google will find its current location).
Use the certmgr to view your certificate and export it. Export it as a certificate (.cer file), and with the key (.pvk) file.
Use cert2spc to convert the cer file into an spc file. That’s the first half of the process done. Now you need a key file compatible with the signcode application.
From http://support.globalsign.net/en/objectsign/transform.cfm:
| How to transform your certificate to a pvk + spc combination.
Export your certificate to a pfx file (be sure to check “Include all certificates in the certification path if possible”). (The latter will help ensure it is accepted by more systems. Install openssl. You can find compiled binaries on www.openssl.org (but get the Windows build from http://www.shininglightpro.com/ http://www.shininglightpro.com/download/Win32OpenSSL-v0.9.7d.exe) Extract your private key from the pfx file. ->openssl pkcs12 -in <pfx-file> -nocerts -nodes -out <pem-key-file> The pfx password will be asked. Download the pvk transform utility. This file can be found at http://support.globalsign.net/en/objectsign/PVK.zip. -> pvk -in <pem-key-file> -topvk -out <pvk-file> Extract your certificates from the pfx file. openssl pkcs12 -in <pfx-file> -nokeys -out <pem-certs-file> The pfx password will be asked. Transform your pem file to a spc file ->openssl crl2pkcs7 -nocrl -certfile <pem-certs-file> -outform DER -out <spc-file> |
I didn’t do that last bit – already did that stage. Note that you are transforming into a PEM file as an intermediate step. I got that bit wrong first time round.
Okay, so now you have all the right bits. You now can just run signcode.exe and use the wizard to check it all works. Select the advanced mode so you can use files, and all should go well. You’ll have to enter the password at various points. If it all worked, then we are nearly there.
To actually sign the code, and I used FinalBuilder to make sure it is done each time and every time reliably, you need the following command.
| c:\codesigning\signcode -spc “c:\codesigning\myAscertiaSPC.spc” -v “c:\codesigning\myascertiakey.pvk” -a md5 -i “www.yourdomain.com” -n “Application install file” “C:\Build\Installer\yourinstaller.exe” |
If it fails, then you probably got a path wrong or a missing file or something silly. Took me ages to spot that I’d got an extra letter in a path. Grr.
The only fly in the ointment is that you need to type the password for each file. It’s not hard to write an app to do the typing for you, but the security of your password is obviously at risk.
An update: Thanks to Hugo Logmans who also provided the script additions to allow FinalBuilder to check that it worked okay, which ensures that you don’t think you signed it.
| BeforeAction: CodeSignFailure = true AfterAction: OnStatusMessage: I have this script to sign the program executable AND singlefile |
Update: After some issues that someone identified with the ascertia certificate on their machine (which I couldn’t replicate), I bought a two year certificate from http://www.instantssl.com/code-signing/ which is from a fairly new certificate authority, but is cheap therefore and is in the XP SP2 root certificates and thus answers the issues.
Update: The codesign executable is a moving target, and is becoming harder to find. So forget the Microsoft codesign executable, and use one that is much easier to use and not only command line compatible but more comprehensive. Give X2Net SignCode a whirl – both GUI and command line options available.
Matthew Jones