The accuracy of dependency analysis in static architecture compliance checking

Abstract

Architecture compliance checking (ACC) is an approach to verify conformance of implemented program code to high-level models of architectural design. Static ACC focuses on the modular software architecture and on the existence of rule violating dependencies between modules. Accurate tool support is essential for effective and efficient ACC. This paper presents a study on the accuracy of ACC tools regarding dependency analysis and violation reporting. Ten tools were tested and compared by means of a custom-made benchmark. The Java code of the benchmark testware contains 34 different types of dependencies, which are based on an inventory of dependency types in object oriented program code. In a second test, the code of open source system FreeMind was used to compare the 10 tools on the number of reported rule violating dependencies and the exactness of the dependency and violation messages. On the average, 77% of the dependencies in our custom-made test software were reported, while 72% of the dependencies within a module of FreeMind were reported. The results show that all tools in the test could improve the accuracy of the reported dependencies and violations, though large differences between the 10 tools were observed. We have identified 10 hard-to-detect types of dependencies and four challenges in dependency detection. The relevance of our findings is substantiated by means of a frequency analysis of the hard-to-detect types of dependencies in five open source systems.