How To Automate Code Coverage analysis with Coverage Lens

Sometimes code coverage is required to finalize the verification. By running only random tests most designs will be left with uncovered sections. Trying to fill up the last bits of code coverage you end up either writing directed tests to reach corner cases, or excluding certain “unreachable” sections from coverage statistics.

Writing directed tests to make sure the DUT reaches a specific state requires manually checking various conditional branches, expressions or FSM transitions across multiple modules. If the design is complex, this can be time consuming.

Excluding unreachable code from coverage statistics without using a formal solution is just an assumption you make. If an assumed unreachable piece of code is actually executed, there should be some failing indication.

Coverage Lens (CL) is an utility (written in C++) that checks if a specified set of RTL code coverage items (statement, branch, condition, etc.) is covered by querying UCIS compliant coverage databases. You can use CL to automate source code coverage checking. You can integrate CL in your regression and let it fire if a set of code coverage items is incomplete.

Table of contents:

How does it work

Flow2

CL takes two inputs:

  • A checks file containing a list of source code entities (statements, FSM states etc.)
  • An UCIS compatible coverage database generated by the simulator

CL searches the code entities specified in the checks file in the UCISDB and it fails if not all of them are covered. It also outputs two reports: one in HTML format and one in plain text format.

A check file consists of multiple “cl_check” commands. Each command describes a specific code entity. For example, to check that a statement is covered you would write a cl_check like this:

cl_check -k inst -p /top/pkt_trans -t stmt -l 332
  • k can be inst or type (inst = check only a specific instance or type = check all instances of a module)
  • p is the instance path in the hierarchy or the type name
  • t is the type of code that we expect to find there (statement, branch, etc.)
  • l is the line

Running Coverage Lens is as simple as:

./coverage_lens.sh --database cover_db.ucd --check my_checks.txt

You can find a detailed description of the checks file syntax and CL arguments in the README.

Examples

How to check that a code entity was covered

/* Simple check file. All checks are for the “pkt_trans” instance from the top module */

#check a statement from line 332
cl_check -k instance -p /top/pkt_trans -l 332 -t stmt   
# first row in an expression’s table 
cl_check -k instance -p /top/pkt_trans -l 333 -t expr 1  
# transition “IDLE->WAIT”  in a FSM named “state”
cl_check -k instance -p /top/pkt_trans/pkt2mac/mac_req_if -t trans state IDLE->WAIT 
# state “SENT” from a FSM named  “state”
cl_check -k instance -p /top/pkt_trans/pkt2mac/mac_req_if -t state state SENT 

Running the above checks file generates an HTML report like this:

Covered

Hovering any line in the table pops-up a tooltip showing the source location.

How to check that a code entity was not covered

CL can also check that items are not covered in the database. You can perform this kind of test by adding a “-n” (negate) argument to any cl_check commands in your check file. For simplicity you can also invoke CL using -n to flip the logic of all checks.

Running the above example on the same coverage databases using -n you get an “inverted” HTML report:

Covered

How to check that waived/excluded code was never executed

You can also invoke CL using waivers generated by simulator tools instead of check files. For waivers, CL will check that the waived code is never hit. You pass the waiver with –refinement/-r .

./run.sh --database UCISDB --refinement waiver

NOTE: CL supports only Cadence’s vpRefine files and Mentor’s “coverage exclude” commands.

How to integrate CL in your regression

To integrate CL in your regression run coverage_lens.sh in the post-run phase of the tests you want to check.

The CL text output looks like this:

*CL_ERR in instance top/pkt_trans,line 333: Condition hs_tx_ready_0 was hit 0 times!
*CL_ERR in instance top/pkt_trans/pkt2mac/mac_req_if/state/: State SENT was hit 0 times!

and you can tune the regression log parser to search for patterns like “*CL_ERR” or “*CL_WARN” in order to detect CL failures.

Performance

CL was built such that it adds minimum overhead to your flow. Running ~4k various checks on a UCISDB of 17k statements, 3.5k expressions, 500 states, and 1200 transitions takes under 0.1s for a minimal single core machine.

Installing Coverage Lens

1. Clone the CL’s GitHub repository.

git clone https://github.com/amiq-consulting/CoverageLens.git

2. Compile CL. CL uses the UCIS API to inspect coverage databases. The implementation of the standard is vendor dependent, so you will have to specify your simulator installation path.

chmod +x compile.sh
./compile.sh {cadence|mentor} [path to simulator install directory]

Running Coverage Lens

chmod +x coverage_lens.sh
./coverage_lens.sh --database some_ucisdb --check check_file

Roadmap

There are few areas that could be improved in the future:

  • Provide support for VCS
  • Provide support for Cadence’s vRefine format

Comments

Ronen July 2nd, 2018 16:34:26

Hi Teo,

Is this tool already support VCS?

Thanks,
Ronen


Stefan Birman July 3rd, 2018 14:28:26

Hello, Ronen,

This application was verified with NCSim and Questa mostly.

If you use it with VCS please tell us about your experience.

Best Regards,
Stefan


Leave a Comment:

Your comment will be visible after approval.

(will not be published)