We are developing at ustwo a very interesting internal little project that needs to run 24/7. By default developers should always be very careful with memory leaks and hogging resources, but with an AIR/Flash app running non-stop every detail counts.

To top it up, the app must handle images, SWFs (AS2, AS3) and videos without going through developers. That means marketing can contact one of the designers, ask s/he to produce someething and put it to display without developer approval. Oh, and all assets are in the network, not in the machine. Scary.

Preliminary tests show we are doing OK, we’ve been running the app non-stop for 4 days without a crash or needing it to reboot. The player takes and releases memory when it needs to, transitions and animations look as smooth as the first day, etc.

Check out below some of our thoughts.

FAIL, FAIL, FAIL

Shit will happen, so be ready to fail. Assets won’t load, video will have the wrong codec, image will be 80mb, network will be down….

That’s Google’s approach to servers. Instead of having a big fat multimillion dollar server, they have a farm of 100 mini-servers and they *know* one or two *will* crash everyday. They’ve build their systems to cope with failure and fail gracefully.

Also embed some default content for when everything else fails.

RELEASE, RELEASE, RELEASE

You must release resources. Explicitly remove every single listener you add, use FP 10 unloadAndStop and if you are in AIR don’t hold references to File objects (more about this in another post).

If you are a good dev, the Flash player will behave most of the times…

MONITOR, MONITOR, MONITOR

…but sometimes it won’t. Or something you didn’t even think was possible will happen in-your-face. You can’t prevent all errors, but you can react when they happen.

At the beginning we were trusting our own app to be its own watchdog, but then we got a nasty crash and obviously the watchdog failed as well. So we’ve created a little haXe/Neko app called “the helper” that is automatically called every minute and that checks the state of our app.

At the moment, the helper is being automatically launched using launchd (fancy cron job for OS X), but that has very interesting side effects (again, more on another post).

The helper checks if our app is running using something like:

And parsing the result for a relevant match. If the app is not running, it starts it again using:

Since we are using AIR 2 beta, we could implement a more refined communication system using the new Native Process API, but for the time being checking if the app is running “the hard way” is enough.

UPDATE, UPDATE, UPDATE

Find a good mechanism to auto-update your app. Most 24/7 apps are unattended apps and they might well be in a different room / building / city / country. But even if you are just a few steps away, you still want to be able to remotely deploy a new version.

We decided to implement our own update system based on GIT commits instead of relying on AIR’s update framework.

LOG, LOG, LOG

Log as much as you can. You’ll be happy you did when you need to find out why your app crashed. Believe me, in this case more is more, even if you face huge log files. If you are a little bit consistent you can always build automatic parsing tools for your logs or just run a simple grep to find error traces.

BTW, we settled for a log file per day to avoid massive log files.

WARN, WARN, WARN

As I said before, most of these apps run unattended, so you want to be notified as soon as possible when there’s an error. You can:

* Use a private Twitter account that people subscribe to.
* Send an email. You can do that from Flash using for example SMTP Mailer and a local SMTP server.

Whatever you do, don’t rely on one method only, specially Twitter, in case is down when the app needs it.

REMOTE, REMOTE, REMOTE

You either enable VNC, SSH or both. If you can’t, then be prepared for walking / driving / taking the bus a lot.

–

I’ll leave the details of running your apps using launchd and the peculiar way in which AIR holds “hard” references to files for another posts.

How is this better than the others?

I think it’s better than Bash because haXe is an OO language and it’s fully crossplatform (no bash in Win unless you use Cygwin). I also think it’s better than Ant because you get rid of bloody-fatty Java as a dependency.

How is this done

In a nutshell: you program in haXe, compile to Neko and then pack to a native app using Xcross. Piece of cake!

Let’s say you need to read some config file, compile your AS using MTASC and copy the output file to several places. Let’s assume a simple config file like this:

And that we pass it to our tool like this:

Now the pseudo-code:

And now compiling:

That should be it; we’ve read and parsed a config file passed as a parameter and then called MTASC handling the result. Then compiled to native Win, Mac and Linux apps.

As I said, that’s pseudo code, on a real life app you would definitely need more error handling or might have a more complex config file (maybe xml, try parsing xml in Bash!), etc., but since haXe is a really similar to Flash or JS, it feels very natural to me. Granted, you can do the same in Bash, but to my eyes is much more complicated.

Some more notes:

• From a Neko app you can call any other tool on your system. That includes SVN, Ant, GIT, grep… Whatever is there, you can call it.
• Apps created with Xcross embed the Neko VM so they have zero dependencies (unless you use any of the tools described above, of course).

We are investigating this route instead of Ant for our build scripts. You might think it’s kind of reinventing the Ant wheel, but we have the feeling that this is going to be much faster and flexible to our needs.

We’ll see : )

PS: I’ve used the xa.* package in the code, that’s a little project I’m working on to provide a simpler haXe API for backends. This is a spin-off of HippoHX and it’s not ready yet, but if you are interested you can follow the development in GitHub.

Our current project at ustwo™ is a PureMVC-multicore application. Sounds really fancy, but it’s only a SWF loading SWFs.

To compile those SWFs we have some scripts using Ant, MTASC, SWFMill, Rhino and other tools. Updating from SVN and compiling 18 modules was taking around 90-100 seconds (that’s updating and compiling the whole project, you can compile single modules much faster).

Keep in mind that our compilation process is a little bit more than calling MTASC. We are running some pre-processing, generating exclude files and some other trickery to gain extra performance (we target mobile devices).

Anyway. Got a hint from one of our developers so I did some research to find a way to speed up compilation time. Since most of us have now dual-core machines we should be able to parallelize some of the work, right? Indeed we can.

Finding PPSS was quite easy but understanding how it to works was a little bit more complicated. I’m not going to bore you to death with the nitty-gritty, so this is the flow that adapted better to us:

* In bash parse the list of module folders and create a txt file containing the path to a script passing to it the path of the module as a parameter. One line per module:

* Feed that txt file to PPSS like this:

That’s where the magic happens. PPSS parallelizes each call to the compilation script using both CPUs. When you run it you can see they go all the way up to 80-90% usage, which is kind of the point.

We took some metrics and found a 40% speed improvement, sometimes even more. If you are a compulsive compiler like yours truly, this saves you quite some time.

Going from the serial approach to the parallel approach wasn’t straight forward. Mostly because I had to split the main script into several scripts and that caused some issues due to my bash programming limitations. This is what I learnt:

* If you execute a script from another, the child doesn’t have access to the variables defined by the parent unless you export them…
* … but arrays don’t get exported.
* Also, you can’t “escalate” exported vars from children to parents. The trick only goes from parent to children (security, I guess).

Anyway. Some more command line black magic under my belt, which is great. The console is a very, very powerful tool that can simplify and standardize daily tasks, which is a must when you are on a team of 10 devs. Not that it is the nicest programming language (actually, it’s pretty ugly), but its ubiquity makes learning it worthwhile.

So, Google a little bit, first result takes me to CLOC.

In theory you should be able to restrict CLOC to a language only but most likely due to my incompetence with regular expressions I couldn’t get it to work the way I liked it. So doing a little bit of command line trickery, I had to create a text file containing the path of the AS files I wanted to parse and then feed that to CLOC. This is how it looks:

Then send that to CLOC:

That would be the simpler option. But because my project is made of modules (it’s a multicore PureMVC app) and the client wanted both stats per-module and global, I had to push it a little bit further. Basically create a stats file per module and then merge them together. Piece of cake:

Where “overall” is the name of the final output file (actually, 2 files are created, one per file, other per language) and “moduleReportX” files are the partial report created before for each module.

Another good thing about CLOC is that it runs in both OSX and Ubuntu at least pretty much out of the box because both come with Perl installed. However, had a little problem in Ubuntu but contacted Al Danial (the author) and he fixed it right away. If you are running Perl 5.10, go to the line 1038 of CLOC and replace “%” by “@”.

Good to go!