The gold standard for a comparison is Tom Schriber and Dan Brunner’s perennial “Inside Simulation Software: How it Works and Why it Matters”. This paper does not attempt to recommend one software product over another, but it does give insight into the inner workings of a variety of simulation packages. When Tom and Dan were including ExtendSim (Extend) into their paper, we exchanged nearly 100 emails detailing the precise behavior of our software. The authors have taken care to update the paper as new versions of ExtendSim have come out. There are other good reviews as well. I have not always agreed with their conclusions, but I do respect their methodology.
I won’t reference the paper on the other end of the spectrum. However, I did find 14 obvious technical errors in a little more than a page of text describing ExtendSim. These included the wrong web site, incorrectly stating that there were limitations on the number of levels of hierarchy, and even the product name was incorrect. The author never contacted us or asked us to comment on his review.
So, if you are looking at simulation software reviews, take the following steps to make sure that you are getting a thoughtful, accurate commentary:
- Contact the author and ask them if the paper was reviewed by the simulation software vendors before publication.
- Look for obvious technical errors. Generally, this is an indication that the author did not do their homework.
- Did they use the latest version of the software?
- Does the author have any connection with a simulation software vendor, currently or in the past?
- If you can, contact modelers who use the software in the paper. Ask if they agree with the conclusions.
As software developers, you can expect us to be biased towards our own creations. Unfortunately, you cannot always find that even in neutral third party evaluations. Caveat emptor.
