GoogleTest lacks certification or qualification as a suitable unit test tool for safety-critical software applications developed under international software safety standards such as ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C. Adherence to safety-critical software standards mandates that tools used for verification activities be certified or qualified. Since Google does not cater to the safety-critical software market, it doesn’t provide tool certification or qualification for GoogleTest.

To utilize existing GoogleTest suites for unit testing safety-critical software, users must consider the following options:

a) Qualify the GoogleTest tool, including every GoogleTest and GoogleMock library macro used.

b) Migrate test cases to a safety-critical standards pre-certified or qualifiable unit test tool.

c) Use the test cases unchanged as inputs, generating relevant test certification evidence through a certified or qualified unit test tool framework. Cantata Hybrid offers this certification engine for Google tests.

Open-source unit test tools (i.e., xUnit tools such as GoogleTest) are not certified or qualified as suitable for use as a unit test tool for safety-critical software developed under international software safety standards (ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C).

All safety-critical software standards require tools used for verification activities to be certified or qualified as suitable for use. Open-source communities do not operate in the safety-critical software market and therefore do not provide tool certification or tool qualification for open-source xUnit tools.

Existing xUnit GoogleTest suites can only be used to unit test safety-critical software if:

a) The GoogleTest tool (i.e., every GoogleTest and GoogleMock library macro used) is qualified by the user.

b) The test cases are migrated to a safety-critical standards pre-certified or qualifiable unit test tool.

c) The test cases are used unchanged as inputs to generate applicable test certification evidence through a certified or qualified unit test tool framework. Cantata Hybrid provides this certification engine for Google tests.

Open-source code coverage tools like GNU Gcov and Lcov lack certification or qualification for use as verification tools in safety-critical software development adhering to international standards such as ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C.

All safety-critical software standards mandate that verification tools be certified or qualified for use. GNU, not catering to the safety-critical software market, does not offer tool certification or qualification for its open-source tools, including Gcov for code coverage analysis and Lcov for coverage reporting.

Commercial vendors, including QA Systems, offer certified code coverage tools such as Cantata and Cantata Hybrid, deemed suitable for verification in safety-critical software development.

The GoogleTest tool is not qualified (or pre-certified) as suitable for use as a unit test tool for safety-critical software applications developed under international software safety standards such as ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C. All safety-critical software standards require tools used for verification activities to be certified or qualified as suitable for use. Since Google does not operate in the safety-critical software market, it does not need to provide tool certification or tool qualification for GoogleTest.

Existing GoogleTest suites can only be used to unit test safety-critical software under certain conditions:

a) The GoogleTest tool (i.e., every GoogleTest and GoogleMock library macro used) must be qualified by the user.

b) The test cases must be migrated to a safety-critical standards pre-certified or qualifiable unit test tool.

c) The test cases must be used unchanged as inputs to generate applicable test certification evidence through a certified or qualified unit test tool framework. Cantata Hybrid provides this certification engine for Google tests.

GoogleTest does not necessarily need to be qualified as a tool for safety-critical software standards such as ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C. Existing GoogleTest suites can be used to unit test safety-critical software if the test cases are used unchanged as inputs, to generate applicable test certification evidence through a certified or qualified unit test tool framework. Cantata Hybrid provides this certification engine for Google tests as an alternative to qualifying the GoogleTest tool.

GoogleTest does not have a qualification kit available for qualifying the tool for use on safety-critical projects. Users can, however, qualify the GoogleTest tool themselves. This involves three activities which can be risky, time-consuming, and expensive:

a) Specifying detailed requirements for all the GoogleTest and GoogleMock macros used in the GoogleTest suites.

b) Conducting tests to demonstrate that all these macros comply with the requirements in the tool operating environment(s) where GoogleTest is used.

c) Maintaining the requirements and test qualification data for each version of the GoogleTest library and for each different tool use environment.

For these reasons, most developers of safety-critical software do not qualify GoogleTest and GoogleMock but instead opt for a certified unit test tool.

Yes, but with a major limitation – only the code coverage results are suitable for use as test evidence, not the checks on expected code behavior. Any commercial C++ unit test tool (including Cantata) or standalone code coverage tool that is certified or qualifiable for use under safety-critical software standards (ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C) can produce certified code coverage results from executing GoogleTest suites.

However, just exercising the code (code coverage) is inadequate evidence of testing for these standards. The test results also need to check if the behavior is as expected. GoogleTest and GoogleMock macros use the uncertified GoogleTest library, so the results of these check macros are not certified or qualified as test evidence.

Only when the GoogleTest macros also invoke equivalent test macros in a certified or qualified unit test tool framework can they generate test check results evidence suitable for certification. Cantata Hybrid provides this certification engine for Google tests.

GoogleTest suites are implemented in C++ using a library of GoogleTest and GoogleMock macros. The input conditions, expected behavior, and outputs of each test case can be converted into the syntax of a unit test framework certified or qualifiable for use under safety-critical software standards: (ISO 26262, IEC 61508, IEC 62304, EN50128, IEC 60880, and DO-178C).

The closer the GoogleTest suite syntax is to that of the certified unit test tool, the easier the task of migrating is. The Cantata tool framework uses a very similar syntax (also implemented and directly editable in C++) to GoogleTest. This makes it viable to manually migrate existing GoogleTest suites to Cantata. However, there is always the risk that data may be incorrectly migrated, and the cost of doing this at scale may be prohibitive.

Cantata Hybrid is an alternative certification engine for large quantities of existing Google tests. The GoogleTest cases are used unchanged as inputs to generate applicable test certification evidence through automatic mapping of the GoogleTest syntax to the Cantata certified and qualifiable unit test tool framework.

ISO 26262 requires that tools used be certified or qualified as suitable for use. Part 8, Section 11, ‘Confidence in the Use of Software Tools,’ defines the means for tool qualification to create evidence that the tool is suitable for use to support the activity or task required. GoogleTest is not certified or qualified as suitable for use under ISO 26262. Google does not operate in the safety-critical software market and therefore has no need to provide tool certification or tool qualification for GoogleTest. Existing GoogleTest suites can only be used for unit and integration testing under ISO 26262 if:

a) The GoogleTest tool (i.e., every GoogleTest and GoogleMock library macro used) is qualified by the user as defined in Part 8, Section 11.

b) The test cases are migrated to an ISO 26262 pre-certified or qualifiable unit test tool.

c) The test cases are used unchanged as inputs to generate applicable test certification evidence through an ISO 26262 certified test tool framework. Cantata Hybrid provides this certification engine for Google tests.

EN 50128 (EN 50657) requires that tools used be certified or qualified as suitable for use. Section 6.7 ‘Support tools and languages’ defines the process for tool qualification to create evidence that the tool is suitable for use to replace manual operations. GoogleTest is not certified or qualified as suitable for use under EN 50128. Google does not operate in the safety-critical software market and therefore has no need to provide tool certification or tool qualification for GoogleTest.

Existing GoogleTest suites only can be used for unit and integration testing under EN 50128 if:

a) The GoogleTest tool (i.e. every GoogleTest and GoogleMock library macro used) is qualified by the user as defined in section 6.7.

b) The test cases are migrated to an EN 50128 pre-certified or qualifiable unit test tool.

c) The test cases are used unchanged as inputs, to generate applicable test certification evidence through an EN 50128 certified test tool framework. Cantata Hybrid provides this certification engine for Google tests.

IEC 62304 is not explicit on the subject of tool suitability, but instead Annex C.7 ‘Relationship to other standards’ refers to the IEC 61508 standard.

IEC 61508 requires that tools used be certified or qualified as suitable for use. Section 7.4.4 ‘Requirements for support tools’ and Annex C defines the techniques and measures for tool qualification to create evidence that the tool is suitable for use to support the activity or task required. GoogleTest is not certified or qualified as suitable for use under IEC 61508. Google does not operate in the safety-critical software market and therefore has no need to provide tool certification or tool qualification for GoogleTest.

Existing GoogleTest suites only can be used for unit and integration testing under IEC 61508 if:

a) The GoogleTest tool (i.e. every GoogleTest and GoogleMock library macro used) is qualified by the user as defined in section 7.4.4 and Annex C.

b) The test cases are migrated to an IEC 61508 pre-certified or qualifiable unit test tool.

c) The test cases are used unchanged as inputs, to generate applicable test certification evidence through an IEC 61508 certified test tool framework. Cantata Hybrid provides this certification engine for Google tests.

IEC 60880 requires that tools used be certified or qualified as suitable for use. Section 14.2 ‘Selection of tools’ defines the criteria and process for tool qualification to create evidence that the tool reliability is verified and assessed as suitable for use. GoogleTest is not certified or qualified as suitable for use under IEC 60880. Google does not operate in the safety-critical software market and therefore has no need to provide tool certification or tool qualification for GoogleTest.

Existing GoogleTest suites only can be used for unit and integration testing under IEC 60880 if:

a) The GoogleTest tool (i.e. every GoogleTest and GoogleMock library macro used) is qualified by the user as defined in section 14.2.

b) The test cases are migrated to an IEC 60880 pre-certified or qualifiable unit test tool.

c) The test cases are used unchanged as inputs, to generate applicable test certification evidence through an IEC 60880 certified test tool framework. Cantata Hybrid provides this certification engine for Google tests.

DO-178C section 12.2, “Tool Qualification,” defines the Tool Qualification Level (TQL) for tool qualification on each specific system. The objectives, activities, guidance, and lifecycle data required are defined in the separate standard DO-330, “Guidance for Software Tool Qualification.” DO-330 section 11.3, “Qualifying COTS Tools,” sets out the detailed process applicable for verification tools as TQL 5.

GoogleTest is not qualified as suitable for use under DO-330. Google does not operate in the safety-critical software market and therefore has no need to provide tool qualification assistance for GoogleTest.

Existing GoogleTest suites can only be used for unit and integration testing under DO-178C if:

a) The GoogleTest tool (i.e., every GoogleTest and GoogleMock library macro used) is qualified by the user as defined in DO-330 section 11.3. b) The test cases are migrated to a DO-330 qualifiable unit test tool. c) The test cases are used unchanged as inputs to generate applicable test certification evidence through a DO-330 qualifiable unit test tool. Cantata Hybrid provides this qualification engine for Google tests.