Evaluating Model-Driven Development Claims with Respect to Quality: A Family of Experiments

Context: There is a lack of empirical evidence on the differences between model-driven development (MDD), where code is automatically derived from conceptual models, and traditional software development method, where code is manually written. In our previous work, we compared both methods in a baseline experiment concluding that quality of the software developed following MDD was significantly better only for more complex problems (with more function points). Quality was measured through test cases run on a functional system. Objective: This paper reports six replications of the baseline to study the impact of problem complexity on software quality in the context of MDD. Method: We conducted replications of two types: strict replications and object replications. Strict replications were similar to the baseline, whereas we used more complex experimental objects (problems) in the object replications. Results: MDD yields better quality independently of problem complexity with a moderate effect size. This effect is bigger for problems that are more complex. Conclusions: Thanks to the bigger size of the sample after aggregating replications, we discovered an effect that the baseline had not revealed due to the small sample size. The baseline results hold, which suggests that MDD yields better quality for more complex problems.