Platform Guide

Verification and Validation

This guide demonstrates how to use Innoslate for verification and validation.

Welcome to the Verification and Validation guide!

This guide demonstrates how to use Innoslate for verification and validation. If you haven’t already, we recommend reading our Requirements Management and Analysis guide before proceeding.

According to the IEEE Standard for Systems and Software, Verification and Validation (V&V) “are used to determine whether the development products of a given activity conform to the requirements of that activity and whether the product satisfies its intended use and user needs.” The goal of V&V is to ensure the resultant product, service, or system being developed meets or exceeds customer expectations. Without implementing proper V&V procedures while engineering a system, it is highly likely the product, service, or system won’t even actually be able to be built, let alone function properly.

Innoslate facilitates these system lifecycle procedures with integrated tools and a new, purpose-built user interface called ‘Test Center.” Our purpose for integrating these tools is to decrease the amount of time it takes to complete the V&V processes, and therefore, encourage those processes to occur more frequently throughout the systems engineering lifecycle.

Validation

Before trying to verify the product, service, or system meets the design requirements and system models, it is important to first validate whether or not those requirements and models are correct. Beginning the validation process early in the lifecycle enhances the probability of success by identifying errors early on in the development phase and preparing for the verification process later in the lifecycle.

  • Requirements Validation

    Requirements validation is an iterative process that should be done throughout the systems engineering lifecycle. Regularly questioning and analyzing the quality of each requirement ensures that at the end of your requirements gathering and capture efforts, you will be left with well-written and verifiable requirements.

    Innoslate provides integrated tools to help automate requirements validation such as the “Quality Check” feature of ‘Documents View.’ The “Quality Check” feature is used to assess a requirement’s clarity, completeness, consistency, design implications, traceability, and verifiability. If you haven’t already, read the “Analyze Requirements” section of our Requirements Management and Analysis guide for more information on the “Quality Check” feature of ‘Requirements View’ and instructions on how to run it.

    During this process, it is important to double-check that each requirement is traceable back to the origin of the requirement and every change has been tracked throughout the life of the requirement. Innoslate takes care of most of this automatically, by creating relationships between entities and the originally uploaded artifact and tracking changes to maintain a version history on every entity.

  • Model Validation

    Model validation is another iterative process that should also be done throughout the systems engineering lifecycle. Regularly questioning and analyzing the quality of each model ensures, at the end of your behavioral and physical modeling efforts, you will be left with a model complete enough to be executable.

    Innoslate provides integrated tools to help automate model validation as well, such as the ‘Intelligence’ tool to assess a model’s traceability, construction, naming conventions, and more. If you haven’t already, please read the “Ensuring Overall Model Quality” section of our Model-Based Systems Engineering guide for more information on the ‘Intelligence’ tool and instructions on how to run it. Other built-in tools include the ‘Discrete Event Simulator’ and the ‘Monte Carlo Simulator’ to help determine whether or not a model is executable and help reduce the uncertainty of a product, service, or system’s cost, schedule, and performance. 

Verification

Now that you are sure that the correct requirements and system models are captured in Innoslate, you can begin the process of verifying the system built meets the previously validated design requirements and system models. This process in Innoslate is primarily accomplished using ‘Test Center,’ a hierarchical document-like display of Test Case entities with collapsible sections, status roll-up, and inline entity editing.

  • Add Verification Method Labels

    Next, you must identify which method or methods of verification are most appropriate to use to verify each design requirement. To capture this information in Innoslate, follow the instructions below:

    1. Click the ‘Database’ button on the top navigation bar to navigate to ‘Database View.’

      database navigation v&v

    2. To filter the view to only display Requirement entities, click ‘Requirement’ in the “By Class” section of the left sidebar.

      requirement filter vandv

    3. Locate the ‘Labels’ in the Metadata Tab on the left sidebar. Determine and select which method of verification is appropriate to verify your requirement(s). Innoslate‘s default database schema provides five different verification method labels: Analysis, Demonstration, Inspection, Modeling & Simulation, and Test.
      label metadata vandv

    4. Then press 'Deselect' on the toolbar and repeat 3 as needed, to add more than one Verification Method label to a requirement.
    5. Click the green checkmark button on the toolbar to save.
      save vandv
    6. Continue identifying appropriate verification methods and adding the corresponding labels until each requirement has a verification method specified.

      database view req filter
  • Download a VCRM Report

    Double-check to ensure each requirement has a verification method label by downloading a VCRM report. Follow the instructions below to download a VCRM report:

    1. Click ‘Documents’ in the top navigation bar to navigate to ‘Documents View.’ Select your document (in this case, “SAE Level-5 Automation Requirements”).

      nav bar documents dashboard

    2. Click the ‘Report’ button.

    3. Select ‘VCRM Output (XLSX)’ as your ‘Report Type.’

      vcrm report selection vandv

    4. Click ‘Create.’

      create vcrm vandv

    5. Once the report has finished downloading, click on the downloaded file to open the VCRM in Excel. An example of the VCRM report is shown below:

      VCRMreport
      1. Create a Test Suite

        The first step in the verification process is to create a suite of tests to eventually run against the product, system, or service. Follow the instructions below to use ‘Test Center’ to create a test suite:

        1. Click to open the ‘MENU’ drop-down on the top navigation bar and select ‘Test Center’ under “General.”

          test center nav menu dropdown

        2. Click ‘+ Create Test Suite’ in the upper right corner.

          create test suite

        3. The ‘Create a New Test Suite’ dialog will appear. Type in at least a name for the root Artifact entity of your new test suite, and then click the ‘Create’ button. In this case, we used the name “Autonomous Vehicle Test Suite.”

          create test suite step 1 vandv

        4. ‘Test Center’ will refresh to display your newly created, empty test suite.

          test suite view vandv

        5. Click the ‘New Test Case’ button.

          add test case in test suite view vandv

        6. This will navigate you to the end of the test suite where a new row has been added. Give your new Test Case entity a meaningful name and be sure to include the procedures necessary to perform the test in the entity’s description or decomposition. In this case, we used the name “Vehicle Steering Test.”

          create test case in test suite vandv

        7. Fill in the ‘Expected Result’ attribute with what conditions are necessary for the test to pass.

          add attribute to test case in test suite view vandv

        8. Click the green checkmark button to save.
        9. Continue adding test cases until you are confident you have identified a procedure to verify each design requirement.
      2. Trace Test Cases to Requirements

        It is important to maintain traceability throughout the verification process so that any engineer reviewing your work months, or even years, down the road can clearly understand what was tested and why. Follow the instructions below to trace your test cases to your design requirements:

        1. Within ‘Test Center,’ click to open the ‘Open’ drop-down and select the ‘Traceability Matrix’ menu item under the “Matrix” section.

          open traceability matrix vandv

        2. Ensure the ‘Top (X Axis)’ is labeled ‘Hierarchy’ in the left sidebar.

          traceability matrix view x axis vandv

        3. Under ‘Root Entity,’ select the name of your requirements document’s root Artifact entity: “SAE-Label-5-Automation-Requirements.csv.”

          select root entity traceability matrix vandv
        4. Under ‘Relationship Type,’ select the name of the relation to compare. In this case, select “verifies.”

          select relationship traceability matrix vandv

        5. Trace which individual Test Case entity from your test suite verifies which Requirement entity from your requirements document by clicking the cell in the matrix where the two intersect. An ‘X’ will appear in the cell indicating a verifies/verified by relationship has been added linking the two entities. You should end up with a matrix similar to the one pictured below:

          TraceTestCases5

        6. Click the ‘Save’ button to persist your changes to your project’s database.
      3. Download an RVTM Report

        1. Click the ‘Documents’ button on the top navigation bar to navigate to ‘Documents View.’

          nav to documents dashboard vandv

        2. Select your Requirements Document: “SAE Level-5 Automation Requirements.”

          select document vandv

        3. From within your Requirements Document, click the Report’ button in the toolbar.

          create report vandv

        4. For ‘Report Type,’ select ‘RVTM Output (XLSX).’

          select rvtm report vandv

        5. Click the ‘Create’ button to generate and automatically download an RVTM report.

          create rvtm report vandv

        6. Once the report has finished downloading, click on the downloaded file to open the RVTM in Excel.
      4. Run a Test Cycle

        Now that you have proven traceability and test coverage, it is time to actually perform all the procedures to run all the tests against your product, service, or system. Follow the instructions below to run your first test cycle:

        1. Click to open the ‘MENU’ drop-down on the top navigation bar and select the ‘Test Center’ menu item under the “General.”

          navigate to test center vandv

        2. From the ‘Test Center,’ select your test suite (in this case, we are using “Level 5 Test Suite” from the example project).

          test center dashboard vandv

        3. Click on the New Test Cycle’ button under the ‘More’ drop-down.

          select new test cycle vandv

        4. ‘Create New Test Cycle’ dialog will appear. For ‘Name,’ type in “First.” Click the blue ‘Create’ button.

          create test cycle vandv

           

          This sets the ‘Actual Result’ and ‘Status’ attributes of every test case in your suite back to blank and “Not Run,” respectively. ‘Test Center’ will refresh to display the changes, as shown below.

          test cycle confirmation vandv

        1. Click anywhere within the first test case’s displayed row to enter its edit mode.

          edit test case vandv

        2. Change the ‘Status’ attribute of that test case to “In Progress.”

          change test case attribute vandv

        3. Click the green checkmark button to save.

          save test case vandv

        4. Using the verification method specified by the test case, perform the test procedure on the product, service, or system.
        5. Fill in the ‘Actual Result’ attribute with your observations of what actually happened as a result of the test procedure.

          add attribute test case vandv

        6. Determine if the test has passed, failed, or otherwise, and then change the ‘Status’ attribute to reflect your decision.

          change status test case vandv

        7. Click the green checkmark button to save.

          save test case vandv

        8. Continue testing until every test in the suite has been performed.
      5. Download a Test Cases Report

        You have executed each test procedure and determined whether each test passed, failed, or otherwise using that test’s specified verification method. You can now download a Test Cases report as proof of verification of your product, service, or system. Follow the instructions below to download a Test Cases report:

        1. From your Test Suite,’ click the Report’ button.

          select report test suite view vandv

        2. Select ‘Test Cases Output (DOCX)’ as your ‘Report Type.’

          select test cases output report vandv

        3. Click ‘Create.’

          create test case output report vandv

        4. Once the report has finished downloading, click on the downloaded file to open the Test Cases in Word. An example of the Test Cases report is shown below:

          TestCaseReport
Now that you’ve been properly introduced to Verification and Validation, we encourage you to further your knowledge by reading our next article: Guided Tours