For those of you who don’t know him, James is an old hand at AI, the person who started RulesFest (which became Intellifest) and a fervent supporter of benchmarking for rules engines. Read his blog, for example here: http://javarules.blogspot.com/2009/03/rulebase-benchmarks-2009.html
He delivered a history-laden assessment of rules benchmarking. Interesting and entertaining.
Benchmarking was absolutely essential a few years ago when the concern in the industry about the scalability of rules engine based solutions still existed. These days the good behavior of these engines under load is relatively well established, and in my experience the benchmarking done has moved from pre-sales and theoretical to post-sales and based on the actual problem being solved.
Benchmarks have value, I agree with James on that, in particular when attempting to compare engine implementations. It’s important, however, to understand what the benchmarks are really measuring. An inference engine has many aspects to it, and the benchmarks tend to be micro-benchmarks that only exercise a few of those aspects. These may or not be exercised in the case of a particular application. As somebody gathering comparative (or not) performance information, you need to extremely aware of this.
An important aspect too is that an application benchmark will exercise more than just the engine itself. As many have found out, it is frequent that in fact most of the performance bottlenecks are in the object marshalling from and to the application. If your marshalling involves serialization and communication, and the objects are large, it’s likely that optimizing that will yield the largest benefits.
Learn more about Decision management and Sparkling Logic’s SMARTS™ Data-Powered Decision Manager
