After a week of tests, we have some results in. I'm hesitant to post the boxplots of our timing measurements because there is an issue we only caught after the fact affecting exactly one PaaS client running on one of our machines. Those tests are being re-run until we have a matching amount of results.
Our control client, a computer designed to handle complex operations like the multiprocess GA, performed essentially as expected. Runtimes for individual populations are tight around the median time, and even the outliers are not egregiously different. One PaaS clearly has more consistent timing than the other, but the less consistent PaaS has faster completion times.
The client representing the Average Joe computer performed about as expected as well. It is clearly slower than the control client, but can still run a whole population in a reasonable amount of time. There are many more outliers on both PaaS plots for this client, but the same trend was seen with one PaaS being more consistent in timing and the other being somewhat faster. One of the PaaS-es also returned the only (legitimate) error codes seen on the plots to this client.
I would talk about the third client, but it's undergoing re-testing on one PaaS, so that discussion will wait until all results are in.
No comments:
Post a Comment