Non-Blocking Socket Communication in SystemVerilog Using DPI-C

As discussed in a previous article (How to Connect SystemVerilog with Python), functional verification may require an interaction between the testbench and components written in various programming languages. The above-mentioned post describes a method for connecting SystemVerilog with Python that assumes a one-to-one relationship between the sent and received packets. This implies that the communication is occurring in a one-way direction (master -> slave), where only one of the connection endpoints is capable of initiating information transfer (SV in this case).

This article shows how you can achieve communication between components written in different programming languages without any dependency between the sent and received packets.

Table of Contents

Architecture

The solution is based on 3 different layers, which are explained in more detail below.

Figure 1.  Connection Overview

Figure 1. Connection Overview

As shown in Figure 1, the three layers constituting the non-blocking communication (the Client does not wait for the Server and vice versa) are:

  1. The User Layer
  2. The Client Layer
  3. The Server Layer

The User Layer

The User layer contains the SystemVerilog testbench that needs to be connected to an external component (represented in Figure 1 as the Server Layer). The testbench interacts with the Server Layer by writing messages to the Client’s input port and reading messages from the Client’s output port.

The Client Layer

To quote Cristian,

The Client layer acts as a proxy between the User Layer and the Server Layer.

As such, a TCP connection with the Server is created from within this layer.

SystemVerilog does not provide native support for TCP sockets, thus some functions used in this architecture are implemented in C++ and called through DPI-C.

Figure 2. Client Overview

Figure 2. Client Overview

This layer is based on a structure called Connection that is used whenever a data transfer is required between the client and the server. This structure has three main functionalities:

  1. Configuration
  2. Sending data
  3. Receiving data

Configuration

First, in order to handle transfers between the two components, there needs to be an established connection through which data can be sent and received. This connection is initialized from the client side by creating a socket and connecting to the server’s IP address on the designated port number.

Once established, the connection is kept active until the end of the simulation, allowing it to be used whenever a transfer takes place. This means that the Client Layer is required to perform the connection configuration only once.

To set up the connection, the user must call the configure() function, passing the server’s IP address and port number as arguments:

// Create connection to server
if(configure(`HOSTNAME, `PORT) != 0)
  $error("Could not establish connection!");

The implementation of the configure() function can be seen in the following code:

// Use this to configure the remote host (the Python server)
// Returns 0 if the connection succeeds, 1 otherwise
extern "C" int configure(const char *hostname, int port) {
  Connection &conn = Connection::instance();

  // ... more code here ... 

  // Try to connect
  int status = 0;
  do {
    status = connect(sockfd, (sockaddr *) &servaddr, sizeof(servaddr));
  } while (status != 0 && status != EINPROGRESS);

  if (status != 0) {
    perror("Connect failed");
    exit(1);
  }

  printf("Connected to %s:%u\n", hostname, port);

  conn.set_state(ConnectionState::CONNECTED);
  conn.set_sockfd(sockfd);

  return conn.get_state() != ConnectionState::CONNECTED;
}

After configuration, all the exported global DPI-C functions operate on a Singleton instance of the Connection structure so that the same connection instance is reused for all operations during the simulation.

Sending data

Sending a message to the server is straight forward: all that needs to be done is to call the do_send() function with the message (as a string) and its length as arguments.

extern "C" int send_data(const char *data, int len) {
  Connection &conn = Connection::instance();
  int result;
  try{
    result = conn.do_send(data, len);
  }
  catch (int i){
    if(i == -1){
      consume_time();
    }
    else if(i == 1){
      printf("\n Error while polling socket! errno = %s \n", std::strerror(errno));
    }
    return -1;
  }
  return result;
}
struct Connection {
   // ... more code here ...  
     
   /**
   * Throws:
   *    -1 on error
   *    1 if sending would block (unlikely)
   *
   * Returns number of bytes sent to remote
   */
  int do_send(const char *data, int len) {

    int status = can_use_connection();
    if (!status) {
      return status;
    }

    int event_ready = poll(&send_event, 1, timeout);
    if (event_ready == -1) {
      throw -1;
    }

    int can_send = send_event.revents & POLLOUT;
    if (!can_send) {
      throw 1;
    }

    int sent = send(sock_fd, data, len, 0);
    return sent;
  } 
  // ... more code here ... 
}

Receiving data

On the other hand, the receive functionality requires a new thread to be started that waits for messages from the server for as long as the simulation is running. This can be done by forking the recv_thread() task after the configuration is done.

// Start recv thread in DPI-C layer
fork
  recv_thread();    
join_none

Calling the recv_thread() task from SystemVerilog triggers an infinite loop that waits for messages from the server through the do_recv() function. This loop is part of the do_recv_forever() function and it only ends at the end of the test.

extern "C" int recv_thread() {
  Connection &conn = Connection::instance();
  conn.do_recv_forever();

  // During the test, the task is enabled, therefore must return 0
  // At the end of the test, the task is disabled, therefore must return 1
  return run_finished;
}
struct Connection {
       // ... more code here ...

  /**
   * Throws:
   *    -1 on error
   *    1 if recv would block
   *
   * Returns number of bytes received from remote
   */
  int do_recv(char *data, int len) {
    int status = can_use_connection();
    if (!status) {
      return status;
    }

    int event_ready = poll(&recv_event, 1, timeout);

    if (event_ready == -1) {
      throw -1;
    }

    int can_read = recv_event.revents & POLLIN;
    if (!can_read) {
      throw 1;
    }

    int received = recv(sock_fd, data, len, 0);

    return received;
  }


  void do_recv_forever() {
    int r;
    char data[BUFFER_SIZE+1];

    // receive transactions forever
    while (!run_finished) {

      try{
        r = do_recv(data, BUFFER_SIZE);
        if (r > 0) {
          data[r] = 0;
          recv_callback(data);
        }
      } catch (int e) {

        // Call to consume_time gives the SV simulator some indication that
        // it can schedule another SV thread for execution. If this exported SV
        // task is never executed, the simulator will continue to poll on the
        // socket for receive, without giving any chance to the send thread to execute.
        // This function is called only when there is nothing to read

        // (1) - means timeout on poll
        if(e == 1){
          consume_time();
        }

        else if(e == -1){
          printf("\n Error while polling socket! errno = %s \n", std::strerror(errno));
        }

      }
    }
  }
      // ... more code here ... 
}

If the do_recv_forever() task does not consume simulation time, the simulator will execute the loop over and over without rescheduling other execution threads. The best way to fix this problem is to consume simulation time when no data is being received. That way the simulator has the opportunity to schedule other tasks. For this reason, the consume_time() task is exported from SystemVerilog to the DPI-C layer.

When a message is received in do_recv(), the recv_callback() function is called in order to save the message in a queue accessible to the User Layer and to trigger an event to notify that something was received.

This function is implemented in SystemVerilog and imported into the DPI-C layer.

Figure 3. Execution flow of simulator threads

Figure 3. Execution flow of simulator threads

The Server Layer

The server constitutes the external component that communicates with the SystemVerilog testbench. After the connection has been configured by the client, the server can process requests from the testbench or initiate a transfer independent of a client request.

The programming language used to implement the server and whether it is blocking or non-blocking is dictated by the needs of the project. For the example project described below, I used a blocking Python Server. The structure of the server is the same as that presented in the article How to Connect SystemVerilog with Python.

Example Project

To illustrate the architecture described above, I created an example project in which the SystemVerilog testbench is connected to a Python server.

To handle the connection between the testbench and the Python server, I created a class (amiq_server_connector) which takes as parameters the IP address of the server, the port, and the character used to delimit items within a message. This class contains a start() function that is responsible for initiating the connection between SystemVerilog and Python and for starting the receive thread. The amiq_server_connector communicates with the testbench through two mailboxes:

  • send_mbox – for sending messages to the server
  • recv_mbox – for receiving messages from the server

The SystemVerilog testbench generates items containing two fields – command and data – which are then sent to the Python Server. Depending on the command received, the server may or may not send a message back to the testbench. Either way, the SV testbench does not need to wait for a response before generating the next item.

The three commands used in this project are:

  • “div”
  • “nop”
  • “end_test”

Upon receiving the “div” command associated with a value, the server sends back a series of messages, each of which contains a divisor of that number. If the command “nop” is received, the server does not send back any response. The “end_test” command is used for the “end of test” mechanism.

Figure 4 shows the data flow for three packets generated in the User Layer. Each packet is received and processed by the Python server according to the command received. The first and the last packets do not have an associated response, since the command is “nop”. The second packet, however, contains a “div” command and a number that has three divisors. Therefore, the Python server will create three messages, each containing a divisor of the number contained in the request. Note that the “SV Send” thread can send packets one after another, regardless of the time required for the Python server to process received packets.

Figure 4. Data flow of the example project

Figure 4. Data flow of the example project

The “end of test” mechanism is based on the sending of a specific command, which, after being processed by the server, generates a response that the client will recognize. When the testbench recognizes this response in the received message, the test is considered finished.

The User Layer for the example project looks like this:

amiq_server_connector#(.hostname("127.0.0.1"), .port(54000), .delim("\n")) client = new();
  
initial begin
  fork 
    // Connect to server and start communication threads 
    begin
      client.start();
    end
      
    // Send thread:
    //   Sends 1000 items to the server through the connector's send mailbox with a random command
    //   Command can be either "div" or "nop"
    //     div: return all the divisors of the number that follows this command
    //     nop: no operation 
    //   The nop command was added as a proof of concept to ensure that the system 
    //   won't be blocked while waiting for a response that is not coming
      
    begin
      string cmd;
      for(int i=1;i<1000;i++) begin
        cmd = get_random_command();
        $display($sformatf("Sending command %s with data %3d \n", cmd, i));
        // An item has the following structure: cmd:value
        client.send_mbox.put($sformatf("%s:%0d", cmd, i));
      end
        
      // End of test mechanism:
      // the last item sent for processing is recognized by the server  
      // after receiving this item the server sends back a particular response
      // which is recognized by the testbench
        
      $display("Sending end of test item \n");
      client.send_mbox.put("end_test");
    end
      
    // Recv thread:
    //    Collecting received items through the connector's recv mailbox
    begin
      string recv_msg;
      forever begin
        client.recv_mbox.get(recv_msg);
        $display($sformatf("Received item: %s", recv_msg));
          
        // End of test mechanism:
        // recognizing the end of test item as a received item
        if(recv_msg == "end_test") begin
          $display("End of test");
          set_run_finish();
          $finish();
        end
      end
    
  join  
end

You can run this example using one of the three major EDA vendor simulators.

  1. Clone the repository using:
  2. git clone "repo" "clone_path"
  3. Export the PROJ_HOME variable to the "clone_path" like this:
  4. export PROJ_HOME="clone_path"
  5. Open the amiq_top.sv file and change the hostname parameter of the amiq_server_connector to "your hostname"
  6. Open another terminal and start the Server with:
  7. python3.6 server.py
  8. Run the arun.sh script with one of the aforementioned simulators:
  9. ./arun.sh -tool {xrun | questa | vcs}

Download

You can download the project from AMIQ’s GitHub repository.

Conclusions

To achieve a non-blocking connection, a receiving thread needs to be started in the DPI-C layer and the simulator needs to be explicitly told when to make a context switch to avoid blocking the simulation.
As the simulation is dependent on an external component, an end of the test mechanism can be implemented based on the server’s response to a specific testbench message.

Enjoy!


Comments

Leave a Comment:

Your comment will be visible after approval.

(will not be published)

This site uses Akismet to reduce spam. Learn how your comment data is processed.