libnetwork

This repository contains an alternative network stack for the T-CREST Patmos Microcontroller, which was initially written as part of this thesis. The goal of the stack was to provide an easy to use but still highly configurable network interface with TSN support for Patmos. In addition, this network stack also provides a real-time capable buffer management implementation based on TLSF.

Structure of the repository

• architecture: contains a UML model of the network stack
• examples: in here some example programmes for the usage of the stack as well as measurement programs for the thesis are stored
• hardware: in this folder a dedicated hardware configure which utilises both network interfaces of the DE2-115 board for Patmos is stored.
• inc: this folder contains the various header files, which shall be used to interface with the library
• libary: this folder holds the actual code for the network stack
• scripts: for ease of use of the examples, this folder contains some helper scripts setting up network applications
• tests: this folder holds some unit tests for hte buffer management

Usage with Patmos

The network stack was developed with the informal requirement to be easy portable to other hardware platforms. Nonetheless, the only platform currently supported is Patmos without an operating system on top. The following descriptions assume a basic knowledge of Patmos. Further it assumed that an environment already set up for Patmos is used, e.g. our developvment container for Patoms see here.

To build the network stack and the example applications run make all inside the repositories root directory. The resulting libary libnetwork.a file will be created in the obj, which will be created during the compilation process. The binary files of the example programs will be created inside the obj/examples folder.

Hardware configuration

To use the TSN Ethernet MAC an advanced hardware configuration is required. The driver requires an additional timer for each interface that shall support TSN. Therefore, we created a dedicated hardware configuration for patmos. The files for it are located inside hardware/patmos. The config folder contains the XML based configuration for the target (altde2-eth.xml). Simply copy it into the hardware/config directory of your checked out patmos repository.

Since the DE2-115 board uses an Altera FPGA, a quartus porject is required as well. The project files are located inside the quartus folder. Copy the altde2-eth folder into hardware/quartus of your checked out patmos repository.

Last but not least the FPGA project requires a top entity. It is located inside the vhdl. This element links the definition of the target with the ports of the FPGA. Copy the patmos_de2-eth.vhdl file into the hardware/vhdk of your checked out patmos repository.

To build the project go to your checked out patmos repository and modify the Makefile in the root directory. Search for the BOARD variable and change it to altde2-eth. Afterwards run make gen in this directory to generate the sources. When this step finished run make synth to synthesize the design. Finaly, run make config to program the FPGA with the design.

Running a program

Once the FPGA is programmed you can download a program. Therefore, go into the root directory of libnetwork and run make patmos. After the command has finished a new directory called obj should exists. In it all build files are stored. Inside obj/examples the built programs are stored, which can be downloaded to the patmos software. Therefore run the following command:

patserdow -v <path to serial interface> <path to binary>

For example:

patserdow -v /dev/ttyUSB0 obj/examples/patmos_udp_hello.elf

Running the WCET analysis

The main reason for the selection of Patmos as the initial platform is its time-predictability. This predictability makes it easier to run WCET analysis. Since platin requires some further information a so-called pml file needs to be generated. It contains certain annotations for the program code like macros defining loop bounds. To create these required files for the example programs run make patmos-wcet. With the following command one can extract cycle values from the created binary code:

platin wcet -i <path to pml file> -b <path to elf file> -e <function name> --report

Key Elements

In the following sections some of the key elements of the network stack are explained

Buffer Management

As one of the major topics inside the original thesis a real-time buffer management implementations are provided. Currently two approaches are implemented:

• Uniform Buffer allocation
• Two-Level Segragated Fits(TLSF)

Where the former simply distributes buffers of the same size, TLSF can distribute buffers of various sizes. The implementations to these approaches are located inside library/buffer_management. For a detailed description on the specific implementation of TLSF please hava a look at Chapter 6.

Drivers

To interact with other network devices the libary provides drivers to interface with the Ethernet MAC in Patmos. There are two implementations availabe, a standard Ethernet driver and a TSN-enhanced driver. The standard Ethernet driver simply takes over the interaction with the hardware, whereas the TSN enhanced driver implements the traffic queues for TSN and the usage of a schedule. The source code for the drivers are located in library/driver. In addition, this folder also contains some boiler plate code to interact with the memory-mapped I/O as Patmos uses special instructions to access I/O devices.

Protocol Stack

The protocol stack provides the functionality for the protocols required for operaiton, e.g. IPv4 and ARP. In general, each protocol implementation provides functionalities to create the required protocol headers and parse the protocol headers.

Services

Are more complex elements which combine protocols and application code. For instance time synchronisation uses PTP and requires a certain amount of internal states, therefore it belongs to the application level. Hence it is implemented as a service inside this stack.

Usage

In the following section we explain the example usage for the socket and network API's. The examples are based on the examples/patmos_udp_hello.c example program.

Configuring a custom memory pool

To use the buffer management within an application, a custom buffer management configuration is required. Therefore, a certain memory pool need to be reserved.

#include "inc/buffer_management.h"
#define MEMORY_POOL_SIZE 0x1000
uint8_t memory_pool[MEMORY_POOL_SIZE];
...
buffer_management_config_t buffer_mgmt_config;
buffer_mgmt_config.start_address = (void *) (memory_pool);
buffer_mgmt_config.end_address =  (void *) (memory_pool + MEMORY_POOL_SIZE)

Currently the buffer management supports two types:

• uniform buffers
• TLSF To configure a memory pool that uses uniform buffers, one simple needs to provide the uniform buffef size.

buffer_mgmt_config.type = BUFFER_MGMT_TYPE_UNIFORM;
buffer_mgmt_config.uniform_cfg.buffer_size = 0x0100;

If TLSF shall be used instead, a configuration shall be provided like the following. The member fl_lowest_cls defines what is the lowest first level class. In this case it is 5, which means that the smallest buffer has a size of at least 32 bytes. Via fl_cls_cnt we define how many first level classes shall exist. In this case we selected 11, since the memory is bounded by 16 bit. Finally, the sl_cls_cntsl_cls_cnt defines the subdivision of the first level. In this case we selected 2. As this value is interpreted as the exponent for a power of 2, for each first level class 4 second level classes exist.

buffer_mgmt_config.type = BUFFER_MGMT_TYPE_TLSF;
buffer_mgmt_config.tlsf_cfg.fl_lowest_cls = 6;
buffer_mgmt_config.tlsf_cfg.fl_cls_cnt = 10;
buffer_mgmt_config.tlsf_cfg.sl_cls_cnt =2;

Since patmos uses typed load and store operation, dedicated memory operations are required to interface with peripherals. Therefore buffer management has dedicated types for memory mapped I/O.

The configuration needs to be handed over to the buffer management such that the mechanism gets set up. Once the setup has finished the function returns an id that shall be used for interacting with it.

uint16_t buffer_mgmt_id = buffer_management_register(&buffer_mgmt_config);

With the malloc and free functions, the user can interact with the buffer management. The former function is used to allocate memory and the latter to return the memory again.

void *buffer = buffer_management_malloc(size, buffer_mgmt_id);
buffer_management_free(void *buffer, buffer_mgmt_id);

Initialising the network stack

Before the network stack can be used it needs to be initialised. When the stack receives a frame, it needs to be temporarly stored until the receiving applications consumes it. Therefore, the stack requires a memory pool provided to it. In addition, the memory needs to be managed, therefore, we use TLSF in this example. Via a call to network_init the buffer management configuration is handed over and the network stack gets initialised for further usage.

#include "inc/network.h"
#include "inc/buffer_management.h"

// Network Stack config
#ifndef NETWORK_STACK_MEMORY_POOL_SIZE
    #define NETWORK_STACK_MEMORY_POOL_SIZE 0xE000
#endif
uint8_t network_stack_buffer[NETWORK_STACK_MEMORY_POOL_SIZE];

...

buffer_management_config_t buffer_mgmt_config;
buffer_mgmt_config.type = BUFFER_MGMT_TYPE_TLSF;
buffer_mgmt_config.start_address = (void *) (network_stack_buffer);
buffer_mgmt_config.end_address =  (void *) (network_stack_buffer) + NETWORK_STACK_MEMORY_POOL_SIZE;
buffer_mgmt_config.tlsf_cfg.fl_lowest_cls = 6;
buffer_mgmt_config.tlsf_cfg.fl_cls_cnt = 10;
buffer_mgmt_config.tlsf_cfg.sl_cls_cnt =2;

// init network stack
network_init(&buffer_mgmt_config);

Configuring a network interface

To configure a network interface certain configuration steps need to be taken in advance. Starting with the basic configuration, for debugging purposes each interface shall have a human readable name.

network_interface_config_t if_config;
    
char eth0_if_name[] = "eth0";
if_config.interface_name = (char*) &eth0_if_name;

The next step of the configuration is to define which interface shall be used. In this case we use the default patmos Ethernet interface. With the option NETWORK_INTERFACE_PATMOS_ETHERNET_TSN the software TSN interface can be selected. The member access_type defines how the driver accesses the interface hardware, in this case memory mapped I/O is selected, since the hardware is directly baked into the patmos softcore. The driver does not require interupts for operation, but it eases it. Since the EthMac Interupt is configured to be on position 6, the interupt ID is 22. To temporarly store frames, a queue is configured by the driver. Via the queue_length member one can configure how many frames can be stored in a queue. Finally, the rx_buffers member define how many buffers are reserved for receiving frames. It is advised to use at least two buffers, such that the Ethernet MAC can store a frame when the other contains already a frame.

if_config.interface_type = NETWORK_INTERFACE_PATMOS_ETHERNET;

if_config.device_config.mmio_config.start_register_address = (void *) (PATMOS_IO_ETH + 0xF000);
if_config.device_config.access_type = DEVICE_ACCESS_MEMORY_MAPPED_IO;
if_config.device_config.interupt_ena = 1;
if_config.device_config.interupt_id = 22; // interrupts start with exception ID 16, EthMac Intterupt = 6 => exception ID 22
if_config.device_config.queue_length = 100;
if_config.device_config.rx_buffers = 5;

The driver uses the buffer management to handly temporarly stored frames inside the Ethernet MAC's memory. In this case we selected TLSF to do this.

if_config.device_config.buffer_mgmt_config.type = BUFFER_MGMT_TYPE_MMIO_TLSF;
if_config.device_config.buffer_mgmt_config.start_address = (void *) (PATMOS_IO_ETH);
if_config.device_config.buffer_mgmt_config.end_address =  (void *) (PATMOS_IO_ETH + 0xE000);
if_config.device_config.buffer_mgmt_config.tlsf_cfg.fl_lowest_cls = 5;
if_config.device_config.buffer_mgmt_config.tlsf_cfg.fl_cls_cnt = 11;
if_config.device_config.buffer_mgmt_config.tlsf_cfg.sl_cls_cnt =2 ;

The next step is to define what kind of layer 2 protocol is used. In this case, we use a Ethernet MAC, the layer 2 protocol is Ethernet as well. Further, we also configure the MAC address in this step.

// layer 2 config
if_config.layer2_config.l2_type = NETWORK_L2_ETHERNET;

uint8_t eth0_if_mac[6] = {0x00, 0x80, 0x6e, 0xF0, 0xDA, 0x42};
if_config.device_config.mac_address.length = 6;
if_config.device_config.mac_address.address[0] = eth0_if_mac[0];
if_config.device_config.mac_address.address[1] = eth0_if_mac[1];
if_config.device_config.mac_address.address[2] = eth0_if_mac[2];
if_config.device_config.mac_address.address[3] = eth0_if_mac[3];
if_config.device_config.mac_address.address[4] = eth0_if_mac[4];
if_config.device_config.mac_address.address[5] = eth0_if_mac[5];

As the last configuration step we set the IP address of the network interface.

// network address configuration
uint8_t eth0_if_ipv4[4] = {172, 16, 0, 201};
if_config.address.type = NETWORK_ADDRESS_IP;
if_config.address.ip.ip_address.version = IP_VERSION_4;
if_config.address.ip.ip_address.ipv4_addr[0] = eth0_if_ipv4[0];
if_config.address.ip.ip_address.ipv4_addr[1] = eth0_if_ipv4[1];
if_config.address.ip.ip_address.ipv4_addr[2] = eth0_if_ipv4[2];
if_config.address.ip.ip_address.ipv4_addr[3] = eth0_if_ipv4[3];
if_config.address.ip.netmask = 24;

To finish the configuration, we need to register the interface with our prevously created configuration. If there exists an issue with the configuration, the funciton will return 0xFFFF.

uint16_t eth0_interface_id = network_register_interface(&if_config);

Continous operation

For the network stack to operate two steps need to executed in a periodic fashion. Updating the interface and reading out frames from the interface. Via network_update the driver gets triggered to send out message and provide new buffers to receive furhter messages. To read out frames, multipe functions exist:

• network_read_interfaces: Read all frames from all registered interfaces
• network_read_interface : Read out all frames from one interface
• network_interface_get_message : Read out a sigle frame from one interface

Configuring a VLAN

For certain usecases it is important to configure a VLAN for the interface, e.g. with TSN. To set up a VLAN set the l2type value to NETWORK_L2_ETHERNET_VLAN. The dei field can be used to mark frames eligible for dropping if congestion occurs. Via the pcp member the default VLAN priority can be set and via the vid member the VLAN ID can be configured.

// layer 2 config
if_config.layer2_config.l2_type = NETWORK_L2_ETHERNET_VLAN;
if_config.layer2_config.vlan.dei=0;
if_config.layer2_config.vlan.pcp=0;
if_config.layer2_config.vlan.vid=100;

Setting up a socket

Setting up a socket is comparably easy like in a POSIX system. Define whyt domain the socket belongs, what type it is and what kind of protocol. In this case we define a UDP socket and since we already defined the Layer 4 protocol (UDP) the last parameter will be ignored.

#include "inc/socket.h"
...
uint16_t socket_id = socket(SOCKET_DOMAIN_IP, SOCKET_TYPE_DATAGRAM, 0);

In this case we configure that directly uses Ethernet. Therefore, we use the raw type, since Ethernet is a point-to-point protocol. In this case we create a socket for a PTP application and therefore we use the Ethertype of PTP for the protocol id.

uint16_t ptp_socket_id = socket(SOCKET_DOMAIN_ETHERNET, SOCKET_TYPE_RAW, ETHERTYPE_PTP);

With the prior socket configuration, a socket can be used to send out data but can't receive data. Therefore, the socket needs to be bound to an address or an interface. In this case wie bind it to hosts eth0 interface and the port 54321.

uint8_t eth0_if_ipv4[4] = {172, 16, 0, 201};
uint16_t src_port = 54321;

...

socket_address_t src_addr;
src_addr.ip_address.version = IP_VERSION_4;
src_addr.ip_address.ipv4_addr[0] = eth0_if_ipv4[0];
src_addr.ip_address.ipv4_addr[1] = eth0_if_ipv4[1];
src_addr.ip_address.ipv4_addr[2] = eth0_if_ipv4[2];
src_addr.ip_address.ipv4_addr[3] = eth0_if_ipv4[3];
src_addr.port = src_port;

socket_bind(eth0_socket_id, &src_addr);

For a socket that uses Ethernet as its transport protocol, the same has to be done, but with the MAC address is shall be associated with.

static uint8_t gptp_multicast_address[6] = {0x01, 0x80, 0xC2, 0x00, 0x00, 0x0E};
socket_address_t ptp_mac_addr;

ptp_mac_addr.hw_address.length = 6;
ptp_mac_addr.hw_address.address[0] = gptp_multicast_address[0];
ptp_mac_addr.hw_address.address[1] = gptp_multicast_address[1];
ptp_mac_addr.hw_address.address[2] = gptp_multicast_address[2];
ptp_mac_addr.hw_address.address[3] = gptp_multicast_address[3];
ptp_mac_addr.hw_address.address[4] = gptp_multicast_address[4];
ptp_mac_addr.hw_address.address[5] = gptp_multicast_address[5];
ret = socket_bind(ptp_socket_id, &ptp_mac_addr);

Sending data with a socket

To send data via a socket, it requires a destination address. In the following example we reuse the prior defined UDP socket. The destination address is configured like the source address.

uint8_t dst_ipv4[4] = {172, 16, 0, 200};
uint16_t dst_port = 12345;

...

dest_addr.ip_address.version = IP_VERSION_4;
dest_addr.ip_address.ipv4_addr[0] = dst_ipv4[0];
dest_addr.ip_address.ipv4_addr[1] = dst_ipv4[1];
dest_addr.ip_address.ipv4_addr[2] = dst_ipv4[2];
dest_addr.ip_address.ipv4_addr[3] = dst_ipv4[3];
dest_addr.port = dst_port;

To send out the data, the socket interface requires the socket id and the destination address. In addition to the data pointer, the interface requires the size of the data as well.

network_error_t ret = socket_send(socket_id, &dest_addr, data, data_len );

The network_error_t is an data type to extract status information form the function calls. The full enumeration list located inside the inc/net_types.h. It should be noted that many of the possible return values are not errors.

Receiving data via a socket

In general, reception is as simple as sending. The receive function requires in addition to the socket id, a buffer and the length of said buffer to validate if the received frame can fit in it. As return values the function, in addition to a status code, returns the source of the message and actual length of the message.

socket_address_t msg_src_addr;
uint16_t data_length;
uint8_t socket_buffer[200];

...

network_error_t ret = socket_receive(socket_id, &msg_src_addr, (uint8_t*) &socket_buffer, &data_length, 200);

Configuring a TSN schedule

The configruation of a TSN schedule consists of three steps:

• defining the schedule entries
• setting the start point
• handing over to the network stack

To definition of said schedule create an array with the number of scheduling entries desired and then configure them one by one. Such an entry consists of two values, the slot duration in ns and the gate state. The gate state is a bit mask. If the transmission gate shall be open, the corresponding bit needs to be set, e.g., if queue 7 shall be open the 7th bit needs to be set i.e. 0b10000000. As further information the schedule shall contain the number of slots and the total duration of it. Further, for configuration checking purposes, the schedule shall contain an entry with the minimum desired guard band. This value is used to verify if the provided schedule is correct.

tsn_gcl_config_t tsn_config;
tsn_gcl_entry_t gcl[4];

tsn_config.entry_list = (tsn_gcl_entry_t*) &(gcl);
gcl[0].gate_state=0x7F;
gcl[0].slot_duration= 1000000; //ns

gcl[1].gate_state=0x00;
gcl[1].slot_duration= 1000000; // ns

gcl[2].gate_state=0x80;
gcl[2].slot_duration= 1000000; // ns

gcl[3].gate_state=0x00;
gcl[3].slot_duration= 1000000; // ns

tsn_config.gcl_length = 4;
tsn_config.cycle_duration = 50000000; //ns
tsn_config.min_guard_band_duration = 1000000; //ns

For the configuration the schedule needs a start point as well. If the start time is set to zero, the schedule will be taken over immediatly. If the start time is set to a time in the past, the schedule will be taken over once the currently active schedule is done. If the start time is somewhen in the future, the already configured schedule runs until the start time comes.

tsn_config.admin_cycle_time.seconds=0;
tsn_config.admin_cycle_time.nanoseconds=0;

To hand over the schedule to network interface, the network_set_option function is used. Its main purpose is to hand over additional configuration to either the stack or an interface.

network_error_t ret = network_set_option(eth0_interface_id, NETWORK_OPTION_TSN_CFG, &tsn_config, sizeof(tsn_config));

Setting the priority value

In TSN communication requires the linking between messages and traffic queues. Therefore, the PCP field inside the VLAN header is used. In the network stack a socket can be configured with a priority value. This priority value does not have an effect on how the frame is treated, except that it is used to map it onto a traffic queue. This value is further used to set the PCP field in the VLAN header.

Via the socket option SOCKET_OPTION_PRIORITY the priority can be set to a value between 0 and 7.

uint8_t value = 0;
network_error_t ret= socket_set_option(eth0_socket_id, SOCKET_OPTION_PRIORITY, &value, sizeof(value));

Setting up time synchronisation service

The time synchronisation service is based on the definitions of PTP and gPTP. The functionality implemented by the network stack takes over the operation for ease of use. The operation is dependent on the configuration. Since we use gPTP which operates directly on top of Ethernet, it requires that an interface is already provided to it, in case multiple interfaces are configured. The various time and interval configuration values are all interpreted as a power of two, e.g. an announceTimeout of 4 means that every 2^4 seconds an announce message will be sent out. These configuration values can also be configured with negative values. The configuration can be configured to select a specific role in a PTP network e.g. a slave.

#include "inc/services/time_synchronisation.h"
...

time_synchronisation_config_t ts_conf;

ts_conf.netw_if = eth0_interface_id;
ts_conf.domainNumber=0;

ts_conf.announceTimeout = 4;
ts_conf.logAnnounceInterval = 2;

ts_conf.logDelayReqInterval = 0;
ts_conf.logDelayRespTimeout = 2;

ts_conf.logSyncInterval = 1;

ts_conf.clock_accuracy = PTP_CLOCK_ACCURACY_2_5_MS;
ts_conf.forced_state = TIME_SYNC_FORCED_STATE_SLAVE;

network_error_t ret = time_synchronisation_init(& ts_conf);

The update function takes over the continous processing of PTP/gPTP messages and processes, e.g. triggering a new round of the delay calculations or sending out announcement messages.

time_synchronisation_update();

Since the system clock of patmos cannot be adapted during runtime, the time itself is a property of the network interface. Hence the current PTP time has to be read out via the options menu from the interface itself.

time_stamp_t eth0_ts;
network_error_t ret = network_get_option(eth0_interface_id , NETWORK_OPTION_GET_PTP_CLOCK, &eth0_ts, &ts_size);
    

Runtime information like the current offset from the master and the mean path delay can be extracted directly from the service.

time_interval_t offset, mean_path_delay;
time_sync_get_runtime_data(&offset, &mean_path_delay);