UNIVERSITY OF VAASA SCHOOL OF TECHNOLOGY AND INNOVATIONS ELECTRICAL ENGINEERING Konsta Mäenpänen TESTING COMMUNICATION RELIABILITY WITH FAULT INJECTION Implementation using Robot Framework and SoC-FPGA Master’s thesis for the degree of Master of Science in Technology submitted for inspec- tion, Vaasa, 4 November 2019. Supervisor Timo Vekara Instructor Jukka Harjula Evaluator Jarmo Alander 2 PREFACE I would like to thank Timo Vekara and Jarmo Alander for the guidance both in the thesis work and during my studies. Furthermore, this thesis was made for Danfoss Drives, so I to thank the company for giving me the opportunity by offering me the topic for the thesis. At Danfoss Drives, I would like to give special thanks to my in- structor Jukka Harjula as well as Petri Ylirinne and Panu Alho, who have instructed and supported me completing this master’s thesis and the preceding bachelor’s thesis. 3 TABLE OF CONTENTS PREFACE 2 SYMBOLS AND ABBREVIATIONS 5 TIIVISTELMÄ 7 ABSTRACT 8 1 INTRODUCTION 9 1.1 Frequency converters 9 1.1.1 Working principle of a frequency converter 9 1.1.2 Control circuit and option boards of a frequency converter 10 1.2 Objective 12 1.3 Structure 14 2 FAULT INJECTION 15 2.1 Serial communication 15 2.1.1 Serial versus parallel communication 16 2.1.2 Synchronous and asynchronous serial communication 16 2.2 Error control in digital communication 18 2.2.1 Types and causes of errors in digital communication 19 2.2.2 Detecting errors in transmitted data 21 2.2.3 Error correction in a communication link 24 2.3 Fault injection testing 27 2.4 SoC-FPGAs 29 2.4.1 Advanced eXtensible Interface 30 2.4.2 Internet protocol and user datagram protocol 32 2.5 Robot Framework test automation framework 35 3 ERROR GENERATOR SYSTEM 40 3.1 Danfoss test automation system 41 3.2 High-speed communication link 42 3.3 SoC-FPGA fault injector 43 3.3.1 Fault injection logic of the SoC-FPGA fault injector 43 3.3.2 Communication between the fault injector and the test library 44 3.4 Test library with Robot Framework 45 3.4.1 Robot Framework keywords for fault injection 45 3.4.2 Manual (targeted) tests 47 4 3.4.3 Automatic tests 47 4 DESIGNING THE ERROR GENERATOR SYSTEM 49 4.1 Design of the UDP/IP communication 50 4.1.1 UDP communication data flow 50 4.1.2 UDP datagram structure 51 4.1.3 Message types for a datagram 53 4.2 Design of the Robot Framework test library 55 4.2.1 Design of the fault injection keywords 56 4.2.2 Design of the setup keywords 58 4.2.3 Designing the UDP handler 61 4.3 Design of the SoC-FPGA fault injector 65 4.3.1 Designing the UDP server 66 4.3.2 Designing the AXI handler 69 5 IMPLEMENTATION OF THE ERROR GENERATOR SYSTEM 71 5.1 Redesign and general notes on the implementation 71 5.1.1 Redesigning the message identifier of the datagram 71 5.1.2 Redesigning the setup of the fault injection library 74 5.1.3 The incomplete implementation of the AXI handler 76 5.2 Implementation of the Robot Framework test library 77 5.3 Implementation of the SoC-FPGA fault injector 83 6 TESTING THE FAULT INJECTOR SYSTEM 86 6.1 Designing the tests 86 6.2 Preparing the testing 89 6.3 Running the tests 91 6.4 Summary and analysis of the test results 93 7 CONCLUSIONS AND FUTURE 94 8 SUMMARY 95 REFERENCES 97 5 SYMBOLS AND ABBREVIATIONS AC alternating current ACK acknowledgement AMBA advanced microcontroller bus architecture API application programmer interface ARM advanced RISC machines ARQ automatic repeat request ASIC application-specific integrated circuit AXI advanced extensible interface BCC block check character CAN controller area network CLB configurable logic block CPU central processing unit CRC cyclic redundancy check DC direct current FEC forward error correction HARQ hybrid automatic repeat request HDL hardware description language HTML hypertext markup language IEEE Institute of Electrical and Electronics Engineers IGBT insulated-gate bipolar transistor I/O input/output IP internet protocol, intellectual property IP/UDP user datagram protocol over internet protocol IPv4 internet protocol, version 4 JTAG joint test action group 6 LCD liquid-crystal display LRC longitudinal redundancy check LSB least significant bit or byte MSB most significant bit or byte NAK negative acknowledgement NRZ, NRZ-I no-return to zero, no-return to zero inverted OSI open systems interconnection PC personal computer PCB protocol control block PL programmable logic PS programmable system PWM pulse-width modulation RF robot framework RZ return to zero SDK software development kit SoC system on a chip TCP transmission control protocol UDP user datagram protocol URL uniform resource locator USB universal serial bus VRC vertical redundancy check XML extensible markup language 7 VAASAN YLIOPISTO Tekniikan ja innovaatiojohtamisen yksikkö Tekijä: Konsta Mäenpänen Diplomityön nimi: Kommunikaation luotettavuuden testaus vianinjek- toinilla – Robot Framework- ja SoC-FPGA-toteutus Valvoja: Timo Vekara Ohjaaja: Jukka Harjula Tarkastaja: Jarmo Alander Tutkinto: Diplomi-insinööri Oppiaine: Sähkötekniikka Opintojen aloitusvuosi: 2013 Diplomityön valmistumisvuosi: 2019 Sivumäärä: 99 TIIVISTELMÄ Taajuusmuuttajia käytetään teollisuudessa laajasti, sillä merkittävän osan teollisuuden sähkönkulutuksesta muodostavat oikosulkumoottorit, joita ajetaan taajuusmuuttajien avulla. Taajuusmuuttajiin on mahdollista kytkeä optiokortteja, jotka lisäävät taajuus- muuttajaan valvonta-, ohjaus- ym. toiminnallisuuksia. Nämä kortit kommunikoivat sar- jaliikenneväylän kautta taajuusmuuttajan pääyksikön kanssa. Sarjaliikennelinkissä, kuten taajuusmuuttajan väylällä, voi syntyä virheitä, jotka häirit- sevät tietoliikennettä. Sen takia sarjaliikenneprotokolliin on luotu virheentunnistus- ja - korjausmekanismeja, joilla pyritään varmistamaan virheetön tiedon kuljettaminen. Luo- tettavuutta testaamaan voidaan väylälle generoida virheitä siihen tarkoitetulla laitteella. Tässä diplomityössä luotiin taajuusmuuttajia valmistavan yrityksen, Danfoss Drivesin (aik. Vacon), pyynnöstä häiriögeneraattorijärjestelmä. Järjestelmä koostuu SoC-FPGA- piirillä luodusta virheitä syöttävästä laitteesta, PC-työkalulle luodusta testirajapinnasta sekä Ethernet-kommunikaatiosta niiden välillä. Laite kytketään väylään, ja testirajapinta tekee testaajalle mahdolliseksi luoda mukautettavia testejä ja ajaa testejä käyttäen Robot Framework -testiympäristöä. Diplomityössä tutkittiin ensin sarjakommunikointiväylien yleisimpiä virheentunnistus- ja korjauskeinoja sekä SoC-FPGA-piirien sekä työssä käytetyn Robot Frameworkin ominaisuuksia. Järjestelmä suunniteltiin ylhäältä-alas-periaatteella ensin tunnistamalla kolmen edellä mainitun komponentin päärakenne päätyen lopulta yksittäisten ohjelma- funktioiden logiikan suunnitteluun. Tämän jälkeen laite ja testirajapinta toteutettiin C- ja Python-ohjelmointikielillä käyttäen suunnitellun kaltaista kommunikaatiota näiden kah- den komponentin välillä. Lopulta järjestelmä testattiin kaikki komponentit yhteen kytkettynä. Varsinainen injek- torilogiikka, joka luo virheitä väylään, ei ollut työn loppuun mennessä vielä toimittavan tahon puolelta valmis, joten järjestelmää ei voitu testata todellisessa ympäristössä. Työs- sä luodut osuudet voidaan kuitenkin myöhemmin kytkeä kokonaiseen järjestelmään. Työn tärkeimpänä johtopäätöksenä on, että tavoitteiden mukainen järjestelmä saatiin luotua ja testattua toimivaksi mahdollisin osin. Jatkokehityskohteeksi jäi mm. kokonai- sen järjestelmän luonti ja testaus oikeaan kommunikaatioväylään kytkettynä. AVAINSANAT: sarjaliikenne, virheentunnistus ja -korjaus, virheinjektointi, häiriöiden generointi 8 UNIVERSITY OF VAASA School of Technology and Innovations Author: Konsta Mäenpänen Topic of the Thesis: Testing communication reliability with fault injec- tion – Implementation using Robot Framework and SoC-FPGA Supervisor: Timo Vekara Instructor: Jukka Harjula Evaluator: Jarmo Alander Degree: Master of Science in Technology Major of Subject: Electrical Engineering Year of Entering the University: 2013 Year of Completing the Thesis: 2019 Pages: 99 ABSTRACT Frequency converters are widely used in industry because a notable part of the industrial electricity consumption is by electrical induction motors driven by frequency convert- ers. It is possible to connect option boards into a frequency converter to add monitoring and control features. These option boards communicate with the main control unit of the frequency converter over a serial communication link. In a serial communication link, e.g. in a frequency converter, it can occur faults that in- terfere with the transfer. Hence, error handling mechanisms are used to secure transmis- sion of the data without errors. A fault injector device, which generates errors into the data travelling in the link, can be used to test the communication reliability. In this master’s thesis, an error generator system was created for a company, Danfoss Drives (previously Vacon), manufacturing frequency converters. The system consists of a fault injector device created with a SoC-FPGA, a testing interface for a PC tool, and an Ethernet-based communication between these two. The device is connected to a serial communication link, and the testing interface makes it easy for a tester to create and run modifiable fault injection tests using a Robot Framework test environment. At the beginning of the thesis, the most common error detection and correction mechan- isms in serial communication and properties of SoC-FPGAs, and Robot Framework were studied. Following this, the system was designed with top-down approach, first identifying the main structure of the components, and finally ending up in designing the logic of individual functions. After this, the device and the testing interface were imple- mented in C and Python using the designed Ethernet communication between them. After the implementation, the system was tested with all the components combined. The actual fault injection logic was not ready by the end of the thesis, so the tests were not run in a real environment. However, the work is done so that the implemented parts can be later used in a complete system. The most important conclusion is that the system was created and tested to meet the re- quirements with applicable parts. Further development includes creating a complete sys- tem and testing it with a real communication link. KEYWORDS: serial communication, error detection and correction, fault injection, er- ror generation 9 1 INTRODUCTION 1.1 Frequency converters This master’s thesis is done for Danfoss Drives which is a manufacturer of frequency converters. Frequency converters have an important role in industry, because a signific- ant share, two thirds of the electrical energy in industry is consumed by electrical mo- tors in industrial applications (Motiva 2006: 14). These motors are mostly induction motors driven by alternating current (AC) from the power grid with a frequency con- verter. The operation speed and torque of an induction motor depend on the voltage and the frequency of the input AC current. Thus, to control the motor, a frequency converter is used to convert the fixed-voltage and frequency AC current from the power grid for the motor energy-efficiently. 1.1.1 Working principle of a frequency converter A frequency converter is used to control the torque and the speed of an electric motor by adjusting the motor input frequency and voltage. It draws alternating current from the power grid and converts it into alternating current that has a desired voltage and fre- quency. Typically, the generated sinusoidal AC current is fed into an electric motor. This alternating current generates rotating magnetic fields which eventually rotate the rotor of the electric motor. (Krishnan 2001: 313). The power conversion part of a frequency converter consist of a rectifier, an inverter and a direct current (DC) link circuit between them. This is depicted in Figure 1. The electric power fed to the variable-frequency drive is normally alternating current with three phases (L1, L2 and L3 in Fig. 1). This alternating current is converted into a DC voltage by using a rectifier which can consist of a diode bridge or thyristors. This DC link is an intermediate circuit between the AC input and the AC output and it holds a DC voltage. Usually, the DC link circuit contains capacitors that reduce ripple in the DC voltage and thus try to keep the level of the DC link voltage constant. (Krishnan 2001: 314–315). 10 Figure 1. The main parts of the power conversion part of a variable-frequency drive. (Hindmarsh & Renfrew 1996: 259). The DC voltage provided by the DC link is then converted by an inverter into alternat- ing current. The inverting is done by the inverter (on the right in Figure 1) that chops the DC link voltage with semiconductor switches, usually IGBTs (insulated-gate bi- polar transistor). These semiconductor switches connect the output of a phase to either the negative or to the positive bus of the DC link (Fisher 1991: 409). The pulse-width modulation (PWM) of the variable-frequency drive controls the length of the pulses so that sinusoidal alternating currents are formed at the three-phase output (U, V and W in Fig. 1), which are needed to operate a rotating electric motor (IEEE 1997: 210–216). 1.1.2 Control circuit and option boards of a frequency converter In addition to the previously described power conversion part, depicted in Fig. 1, the variable-frequency drive also has a control unit (control circuit) that drives the logic of the device. It controls the inverting circuit by switching on and off the semiconductor switches to produce the desired waveform for the output. The control unit is also re- sponsible of receiving inputs such as adjustments in speed or torque or to stop or start the motor, and to react to them accordingly. The main parts of a control unit are shown in Fig. 2. (Danfoss 2000: 52). 11 Figure 2. Main parts of a variable-frequency drive, including the power conversion part (rectifier, DC link and inverter) and the control circuit. (Danfoss 2000: 52). The control unit communicates also with the user interface logic. The user of the drive can perform the operations by a physical operator panel on the front side of the variable- frequency drive, shown in Figure 3, or by a personal computer (PC). Figure 3. The operator panel of a Vacon NXP frequency converter with an LCD (li- quid crystal display) and a keypad. In addition to this, the control unit handles other digital and analogue inputs or outputs which may be connected to an option board that performs a special task. These exten- sions can be inserted into the existing slots inside the variable-frequency drive to add functionality without replacing the variable frequency drive or its control unit. The op- tion boards can be designed to handle some special operation and communicate with the drive itself. (Danfoss 2016: 4–5). 12 Option boards can be connected into a variable-frequency drive to easily add functional- ity. Each card has a dedicated functionality, and they communicate with the control unit of the drive through a serial communication link. This communication between the op- tion boards and the control unit must be robust to ensure that the functionality works as intended, which is critical especially in safety-critical applications. Thus, serial commu- nication links need to make sure that instructions are passed between the components without errors. The robustness of the communication is a vital part of the functionality of a variable-fre- quency drive. Therefore, verifying how the error handling in the communication is working, cannot be overlooked. To improve testing process effectiveness and the cover- age of the tests, proper development procedures such as test-driven development and different testing frameworks can be utilised. 1.2 Objective The objective of this thesis is to create a system generating errors, an error generator system, depicted in Fig. 4. Its task is to inject faults into a serial communication link to test the reliability of the communication. The created error generator system is used in the test automation system of Danfoss. The system consists of the components that are depicted in Figure 4: 1. A fault injector, that is an SoC-FPGA (system-on-a-chip field-programmable gate array). It generates and injects errors into a communication link connected to it. 2. Test library, which provides an interface for a tester to create and run tests with a personal computer (PC) by using the fault injector. It consists of fault injection functions (“keywords”) that can be called, and test templates. The test library communicates with the fault injector over Ethernet. 3. A high-speed communication link and the nodes communicating over the link. The fault injector is connected to the link, and the tests that are generated and run with the testing library should generate errors in the data in the link. 13 Figure 4. The initial schematic of the error generator system that is designed and im- plemented in this thesis. The objective of this thesis is divided into three parts: 1. Designing and implementing the Robot Framework test library (Robot Frame- work is discussed in detail in Subchapter 2.5). 2. Creating the design for the programmable system (the processor) of the SoC- FPGA fault injector. Firstly, it communicates over Ethernet with the test library, as well as with an AXI (Advanced eXtensible Interface) protocol with a fault in- jection logic block. 3. Designing the communication between the test library and the SoC-FPGA fault injector over Ethernet. The fault injection logic block is an intellectual property (IP) FPGA block, that is imple- mented on the same SoC-FPGA as the fault injector. It does the actual error generating and injection on the communication link, that is connected to the SoC-FPGA. However, the fault injection logic block is designed by Danfoss, and thus it is supplied later for the thesis and designing it is not included in the objective. Also, the high-speed communic- ation link, the nodes on the link and the data transferred in the link are not included in the objective of this thesis. The communication link and the fault injection logic block are shortly discussed more in Subchapters 3.2. and 3.3. 14 1.3 Structure The structure of this thesis consists roughly of three larger parts: the theory, the design and implementation and the testing part. In the theory part, essential theory is studied to provide the possibility to continue with the implementation and especially the testing part where the results are reflected with the presented theory. The implementation part describes the creation process of the system, and finally, testing part lists and analyses the test results and other observations made while demonstrating the device. A significant part of Chapter 2 concentrates on research of the serial communication protocols. They are studied focusing on error detection and error handling techniques. Since the underlying communication bus protocol is implemented by the company, its implementation details are not shared. However, the error handling mechanisms of two other protocols are shortly described to give reference. SoC-FPGAs and the Ethernet communication possibilities are also studied with some extent to give base and helpful reference for the design and implementation phase. Finally, the test framework, Robot Framework, is introduced, focusing on what can be achieved with it. The design and implementation phase starts with the more detailed description of the structure of the error generator system. All the parts of the error generator system that are created (the SoC-FPGA fault injector, the Robot Framework test library and the communication between them) are described, including how the system is integrated as a part of Danfoss test automation system. After this, both the hardware part on the SoC- FPGA and the testing library with Robot Framework are designed. Finally, the design is implemented and the implementation details are given. Chapter 6 includes the planning of tests, the description of the test environment as well as the test results. The test results are analysed and summarised at the end of the chapter by using reflection from the theory and the objective parts of the thesis. Finally, the pre- vious parts are wrapped up with conclusions and a summary. 15 2 FAULT INJECTION In this chapter, required theory is studied to give reference and help for the work in the design and implementation phase as well as the testing for the error generator system. The study tries to flow logically from the communication link, where the errors are gen- erated and injected, to the software where the tests are eventually run, opening all the relative theory needed in between. First, we start by introducing the serial communication basics and their error detection techniques to understand what is happening inside the error generator system. Then some research is done on fault injection in general as well as faults in digital communic- ation to grasp consequences of faults and what benefits can be achieved by using fault injection in testing and increasing robustness of a system against those faults. After this research, which is carried out in rather a lecture-like form, the remaining study focuses on the practical part of this thesis. Study is done on how the hardware of the SoC-FPGA fault injector works and how it communicates with other components in the error generator system. Ultimately, introduction of a test automation framework “Robot Framework” which is used in the thesis for the test library, is given. 2.1 Serial communication Serial communication in general consists of sending data in serial rather than in parallel. In other words, in a serial communication, a sender sends, and a receiver receives one bit after another instead of multiple bits at the same time (Axelson 2007: 11). To under- stand the effects of fault injection into a serial communication link, the basics of serial communication are studied. In this subchapter, the focus is on the physical layer and the data link layer, the two bot- tom-most layers, of the OSI (open systems interconnection) model. OSI model is a con- ceptual model defined by an ISO (International Organization for Standardization) stand- ard. According to the model, communication is built on layers, each of which provides 16 defines functionality for the layer above, by encapsulating the functionality from the layer below (ISO 2002). 2.1.1 Serial versus parallel communication Serial communication is used very widely in different applications, mainly due to its more efficient usage of the data transfer medium than parallel communication methods. When a serial communication is used, in the most optimal situation only two wires (one for the data and one for ground) can be used to transfer the data when an electrical transfer medium is used. There might also be a clock signal if the communication is synchronous. A serial communication without a clock signal is called asynchronous communication. (Dell 2015: 90). In a parallel communication, more wires are needed to send the data. For example, sending an eight-bit byte requires at least eight parallel data wires and a ground wire. In general, parallel communications are easy to implement and in addition, because more data can be sent over a parallel communication during the same time, they are usually faster than serial communications. This is also a drawback, because it increases the manufacturing costs of the cables and the connectors and using more signal paths in an integrated circuit reduces the available space in the circuit. Furthermore, due to the large number of conductors, there is more interference and crosstalk in parallel communica- tion than in a serial link. Also, because of this, the transfer distance is limited which is why serial communications are mostly preferred and used especially when data is sent over longer distances, e.g. from one circuit to another using a cable. (Murthy 2009: 119–120). 2.1.2 Synchronous and asynchronous serial communication In a serial communication – or in digital communication in general – it is vital that the receiver and the transmitter of the data are synchronised, meaning that the received data from the transmitter is sampled at the correct interval at the receiver. If the parties were not synchronised, there would not be a way to determine, where the data would start and end, resulting into corrupted data and inability for the parties to communicate. These synchronisation errors in critical applications, such as in a variable-frequency motor 17 drive, could lead to fatal consequences. Thus, here, a comparison of synchronisation methods is done to grasp how the synchronisation can be carried out and what are their effects on both the communication robustness and transfer speed (Dell 2015: 90). As shown in Figure 5, the synchronisation can be carried out in a protocol by a separate clock signal or by adding information in the data to provide the receiver the possibility for synchronisation. Figure 5. An illustration by Axelson (2007: 13) on how synchronous serial commu- nication requires a separate line for the clock, whereas an asynchronous transmission needs synchronisation bits (here a start bit and a stop bit) to synchronise the data. In a synchronous transfer, a clock signal is transferred alongside with the sent data. This pulsating clock signal states the beginning of each data bit which synchronises the communication. With this setup, high speed rates can be achieved. The drawback is that transferring the clock signal requires an additional wire and the protocol must define who the master is, i.e. which party of the communication is responsible of generating the clock signal. (Murthy 2009: 122). 18 In asynchronous communication, however, no clock signal is being transferred. The transmitter and the receiver must agree beforehand on the speed of the communication, e.g. by a definition in the standard or by configuring the parties. Despite this, because of inaccuracy, the clocks of the receiver and the transmitter may and will probably not be exactly the same. Therefore, by sending synchronisation bits at the beginning of each data packet (a transfer unit of data consisting of bits), the receiver can both acknow- ledge when the actual data starts and synchronise the clock if any skew has occurred in relation to that of the transmitter. This way with asynchronous communication, one wire for the clock can be saved, as well as the configuration for the clock master. On the other hand, extra data must be sent in form of synchronisation bits which increases the overhead. Furthermore, another drawback is the clock skew which limits the maximum speed of the communication. (Axelson 2007: 11–12). 2.2 Error control in digital communication In this thesis, an error generator system is created to test the tolerance of the communic- ation link against faults. With a proper error detection and handling in this serial com- munication protocol, number of errors can be decreased. As stated, the system that is created, tries to simulate real-life faults and errors by inten- tionally generating and injecting them into the communication link to test robustness. These real-life faults and errors in digital communication can be caused by multiple factors “in the field”. As stated in Subchapter 2.1.2, inaccurate or flawed clocks can per- turb the communication if synchronisation is not implemented properly. In addition, outside factors such as interference and disturbance in the actual transfer medium are simple reasons for an error. Furthermore, the limited capacity of a communication link can cause errors if congestion in the communication occurs. The causes of faults are dis- cussed in more detail and listed in Subchapter 2.2.1. In general, error control against these faults in any protocol consists roughly of two parts: managing how the receiving end should detect an error in the message (error de- tection) and the operations to ensure that the correct message can be delivered to the re- cipient by the protocol should there be an error (error handling). To comprehend the 19 fault management in the communication link, different error detection and correction mechanisms are studied and compared in Subchapters 2.2.2 and 2.2.3 to provide base for reflection in the eventual testing phase. 2.2.1 Types and causes of errors in digital communication As stated earlier, the error generator system simulates real-life faults on a serial link. These errors can occur in several ways. Firstly, through the electromagnetic interference (EMI), disturbing currents can be induced in the electric transmission link, modifying the digital signal travelling in it. The electric transfer medium can either absorb electro- magnetic interference from other electric devices that are sources of electromagnetic fields, or it can be interfered with by another nearby communication wire, also known as crosstalk (Malarić 2010: 69–75). The interference can come from a natural source, too. A lightning strike or electrostatic discharges are typical examples of this (Malarić 2010: 59). Secondly, all electrical signals contain some amount of noise. It is a random disturbance in a signal, and it is caused by e.g. the thermal motion of the electrons (thermal noise) as well as the randomness in their movement (shot noise). Usually, the noise level is low but if the relative amplitude of the noise in a signal rises too high compared with the ac- tual data signal, the information in the signal cannot be distinguished from the noise. (Malarić 2010: 58). Furthermore, as already stated in Subchapter 2.1.2, the lack of synchronisation can cause errors, too. If the communication protocol does not define enough means of re- synchronisation of the transmitter and the receiver by for instance a clock signal or a self-clocking line coding, clock skew can occur, meaning that the receiver samples the signal at an erroneous point, which can possibly lead to a faulty bit value read by the re- ceiver. Synchronisation errors can occur even if there were no interference or disturb- ance with the travelling in the communication link. Errors in the digital communication are realised as flipped bits. This means that the transmitter intended to send a bit but because of an error in the signal, the receiver reads 20 the value of the bit flipped, so that the value of the bit is of opposite polarity. Single and multiple bit errors are shown in Fig. 6. Figure 6. Single bit errors and multiple bit errors. If only one bit in the data packet changes, a single bit error occurs. However, in the serial communication protocols, the duration to transfer a single bit is very small. There- fore, it is very unlikely that only a single bit is flipped because of an error, so single bit errors are relatively rare in serial communication. Thus, it is more likely that multiple bits are flipped because of a fault. This is because the erroneous state in the communication link usually lasts for a relatively long time compared to the transfer speed. This is called a burst error. Multiple bits – which do not necessarily have to be consecutive – are flipped, causing incorrect data. If the erroneous state lasts very long, the whole packet can be corrupted by flipped bits, causing that the other end will never receive the packet, resulting in a lost packet. Furthermore, if the communication protocol uses automatic repeat request, ARQ (dis- cussed later in Subchapter 2.2.3), the receiver must send information to the transmitter that it had received the data. If an error occurs while sending this acknowledgement, the transmitter would try to resend the same data. In this case, a duplicate packet error takes place, because the given packet has already been sent. Data packets can be lost because of an error or they can intentionally be dropped, lead- ing into dropped packets. This can happen if there is too much traffic on the communic- 21 ation link. If too many packets would be sent into the communication link, the transmit- ter has to drop some of the packets to avoid traffic congestion in the communication link. 2.2.2 Detecting errors in transmitted data Detecting errors in communication protocols is based on sending data with inserted ad- ditional or redundant information that can be used to validate data integrity at the re- ceiving end. In a very simple example, the protocol can define a following error correc- tion mechanism: If the sender sends four bits, e.g. 0110, it should add those four bits re- peated as a data validation field, thus sending 8 bits (0110 0110). So, if the protocol is followed, when the receiver receives anything different than a packet that consists of two exactly same four-bit bytes, it will detect an error and the communication will begin a procedure to handle the error. The principle is simple. However, because the given ex- ample would be very slow and ineffective, much more accurate and effective algorithms are used in practice to calculate the additional error detection field, some the most com- mon of which are presented next. Parity is a simple way to check that the data is transmitted immutably. Parity check is used to validate data in a byte level by adding a parity bit at the end of the actual data bits. Thus, if a byte contains an even number of 1’s, the parity bit should be zero. Con- versely, if the number of 1’s in the byte is odd, the parity bit is set to one. This is called even parity, since an even number of 1’s produces a 0 parity bit. Hence, odd parity would mean that the value of the parity bit is inverted. By checking if the number of 1’s in the data byte is the same as stated by the parity bit, an error can be detected. How- ever, if an error caused an even number of changes in the data bits, the error would go undetected. For example, noise bursts usually cause disintegration to more than only one bit, which is why parity is not very reliable form of detecting errors. (Hioki 2001: 519–522). Previously mentioned way of calculating parity is usually referred as vertical redund- ancy check (VRC) because the parity bit is calculated for each byte. Parity bits can be computed also at the end of the block of a multiple-byte-message, which is called lon- gitudinal redundancy check (LRC). The parity is computed for each bit position from 22 LSB (least significant bit) to MSB (most significant bit) and the possible VRC bit. With the combination of VRC and LRC, the position of the erroneous bit can be found and corrected unlike if only vertical parity was applied. However, like in vertical redund- ancy check, an even number of errors in the bits cannot be found. (Hioki 2001: 522). Table 1 gives an example how the LRC and VRC are calculated for a multiple word message. Table 1. A message consisting of four 4-bit data words. Each data word has in addi- tion a parity bit calculated with vertical redundancy check. A parity calcula- tion is then applied to each bit position (from bit 0 to VRC) into the LRC column. Here, even parity is applied. Word 1 Word 2 Word 3 Word 4 LRC bit 0 0 1 1 0 0 bit 1 1 1 0 1 1 bit 2 0 1 0 1 0 bit 3 1 0 0 1 0 VRC 0 1 1 1 1 Cyclic redundancy check (CRC) is a more effective way to detect errors. In the CRC, the message block is divided by a generator polynomial and the remainder of the divi- sion, a block check character (BCC), is appended at the end of the message block, which is depicted in Fig. 7. The receiver then checks the integrity of the data by divid- ing the message block containing the block check character by the same generator poly- nomial: expected remainder of the division should be 0 if the data was transferred without any errors detected. (Peterson & Brown 1961: 228–235). 23 Figure 7. An example of creating the BCC using CRC according to Hioki (2001: 526). Here, the divisor is the generator polynomial X5 + X2 + X + 1 = 100111 and the dividend is the actual data 101001101 with 5 trailing zeroes. The number of zeros added defines the length of the BCC. Because CRC is more efficient than the vertical and longitudinal redundancy checks and still does not increase the implementation costs significantly, it is used widely as the main error detection mechanism in many serial communication protocols. (Hioki 2001: 524–525). Checksums are another popular way to detect errors. As the name implies, the bytes in the data are added up and the sum is appended at the end of the actual message as the block check character (BCC). Depending on the length of the BCC, the checksum can be a single-precision checksum or a double-precision checksum. Here, any overflow or carry is ignored; the accuracy of the error detection can be increased by using either the residue checksum where the overflown value is added back to the checksum or the Hon- eywell checksum where the checksum is calculated by summing up the bits of two con- secutive data words instead of one. (Hioki 2001: 529–531). Cyclic redundancy check is a very common fault detection mechanism, thanks to its high error detection rate. For example, universal serial bus (USB) which is a popular serial communication protocol commonly used to connect peripherals and other devices with a personal computer, uses CRC to detect corrupt packages. Another popular serial communication protocol, controller area network (CAN), which is used broadly in auto- 24 motive industry, relies its error detection on CRC by, similarly to USB, appending a CRC field at the end of each data packet sent as shown in Fig. 8. (Compaq & al. 2000: 1999 , Bosch 1991: 59). Figure 8. CAN frame format showing the CRC and ACK fields that are vital for error management (Davis & al. 2007: 245). 2.2.3 Error correction in a communication link After an error is detected, the protocol should handle how to manage it. There are two basic types of correcting the errors in the communication; automatic repeat request (ARQ) and forward error correction (FEC). In ARQ, the receiver must automatically send a response to the sender acknowledging that the received data is transmitted cor- rectly. If the receiver detects an error by using an error detection mechanism, e.g. a CRC, it should respond the transmitter negatively or not at all so that the transmitter may try to send the data again (Hioki 2001: 532). The example of a sequence of frames sent with an ARQ is shown in Fig. 9. Figure 9. The sender sends three data frames (F1, F2, F3) with an ARQ protocol. The receiver detects an error in frame F2 and sends a negative acknowledgement (NAK) to force the sender to resend the frame. 25 In FEC, however, the sent data includes a correction code which injects redundant bits in the data. The receiver can decode the correction code then to both detect the error and to correct the error (Hioki 2001: 532). This decreases the need to retransmit the data if an error occurs, which is why FEC is preferred in communications where resending the data would be too expensive or not possible. An example of a FEC mechanism is a very widely used hamming code (Hamming 1950) but the implementation details will not be presented here. One of the previous two can be applied for a protocol. In addition, a combination of these two can be used to both request a retransmission and correcting errors by a code. This is called hybrid automatic repeat request (HARQ). In HARQ, in a simplified man- ner, the receiver tries to correct errors with the error correction code. If there are too many errors, an automatic repeat request is sent. (Wicker 1995: 409). Another possibility is blind transmission, or unacknowledged connectionless transfer, where neither positive or negative acknowledgement is returned to the transmitter. Hence, the transmitter cannot be sure whether the transmission was error-free or whether the data ever received its destination. Even though data transmission cannot be re-requested, data with faults are ignored (“dropped”) by the receiver e.g. with the CRC, so that only correct data will be handled. (Shinde 2000: 171). The obvious drawback of blind transmission is the uncertainty in the transmission, but not acknowledging the data reduces the bandwidth usage of the communication link and increases speed, when there is no acknowledgement and retransmission. The method is useful, when the bandwidth is limited, acknowledgement is not necessary or when there is no possibility to acknowledge (lack of return channel). To reduce the number of lost packets, the transmitter can send a packet multiple times and “hope” that at least one of them will receive the destination. Table 2 shows the comparison of the error correction mechanisms studied here. As can be seen, all the correction mechanisms have some advantages but also drawbacks, meaning that the mechanism should be carefully selected to fit the application. 26 Table 2. Error correction mechanisms in a nutshell. Error cor- rection mechanism Basis of correction Advantages Drawbacks ARQ Positive or negative response returned depending on the result of error detection Data is sent reliably ACKs use band- width and retrans- missions cause delay FEC Correction code included in data; receiver detects and corrects the error No need to return acknowledgement Additional correc- tion code must be inserted into the message HARQ Combination of ARQ and FEC Best of both ARQ and FEC Those of ARQ & FEC Blind trans- mission Data with errors is dropped but no repeat request Speed and low use of communication link. No additional codes. Lost data cannot be recovered The previously presented serial communication protocol examples, USB and CAN, both rely on ARQ as their error correction mechanism. In USB, the client should send an ac- knowledgement packet (ACK) back to the host after each data packet that is sent, if no errors were detected. If the host does not receive an ACK packet within a certain time limit, a frame, it will assume that the transmission failed and will resend the same data packet. In CAN, however, because of its distributed nature, if none of the receiving devices detects that the transmission was error-free, an acknowledgement error occurs, and the sender will try to retransmit the packet. Furthermore, if a device detects an error, it will send an error frame, which corrupts the packet that is being transmitted and forces the sender to retransmit. With the acknowledgement mechanism in the ARQ based error correction, the corrup- ted packets in USB (or frames in CAN) that are detected by the CRC can be resent properly to the receiver. Furthermore, acknowledging each packet ensures protection against lost packets; if the packet is lost, the transmitter will not get an ACK response and is forced to resend the packet. The ARQ must be extended to detect duplicate packets. In USB, for example, detecting a duplicate packet is implemented by data toggling, i.e. adding a packet identifier, that is 27 toggled between DATA0 and DATA1 for each sent data packet (Axelson 2015: 51–53). A duplicate packet would be detected by the USB device if the packet identifier did not change between consecutive received data packets. On the other hand, because the transmitter in a CAN bus doesn’t communicate directly with the receiver but – unlike in USB – it broadcasts the frame to all the receivers, managing duplicate packets is not done by the protocol itself but should be implemented by the receiving nodes. 2.3 Fault injection testing In this thesis, an error generator system is created to test communication reliability with fault injection. In fault injection in general, a set of selected faults is injected into a sys- tem. The system is then monitored to validate that the response of the system matches the defined specifications during faulty conditions. In this way, test coverage can be im- proved when also more seldom error conditions can be tested, too. According to Benso & Brinetto (2003: 28–30), the faults in a hardware system can be divided into three main categories: • hardware faults caused by wearing out or breaking of a hardware component, • intermittent (or periodic) faults because of the instability of the system, and • transient faults from e.g. EMI and noise. So, in our error generating system, it can be validated that the communication link (the system) manages the corrupted or lost packets (the injected faults) by detecting the er- rors and starting the error management procedure defined in the protocol (responding by the definition). (Koren & Krishna 2007: 355, Benso & Brinetto 2003: 35). The permanent faults are relatively infrequent and easy to detect, and the hardware sys- tem can be recovered from them by repairing or replacing the component that causes the fault. Also, the system can be recovered from intermittent faults by either component re- placement or redesigning the system if the intermittent fault is caused by a design error (a “bug”) (Benso & Brinetto 2003: 28–29). 28 On the other hand, the transient faults are the most common faults and they are more difficult to detect than permanent or intermittent faults. Usually, the transient faults do not cause significant damage to the hardware system itself but rather leave the system into a faulty state for a short time which is likely to cause unexpected behaviour in the system. Thus, fault injection tests on hardware, e.g. on a communication link, should have their focus on the transient faults. (Benso & Brinetto 2003: 11–12). It is possible to generate transient errors that resemble the effects of a real-life fault dir- ectly in the hardware. However, direct hardware fault injection can damage the system and it is very device-specific and thus not a portable solution (Benso & Brinetto 2003: 30–31). A more popular way is to connect a software-based fault injector device into the hardware system. The software in the fault injector manages changing the bits in the data, producing a fault with same effects as an actual physical fault (Koren & Krishna 2007: 357). So, the SoC-FPGA fault injector in this thesis can be connected on the communication link between the parties. It manipulates the data sent by the transmitter before its trans- mitted to the receiver, and so it injects a fault in the communication system that reminds of a real hardware fault, such as excessive electromagnetic interference on the commu- nication link. With the error generator system, the following faults are able to be injec- ted into the communication link with the use of the Robot Framework test library: 1. CRC fault: bits in the message data are flipped so that the CRC calculation is vi- olated. CRC field is not modified. 2. Change target address: The message contains a “target address” field. Its con- tents are manipulated so that the message is received by an undesired receiver. 3. Increasing/decreasing the length of the message. 4. Duplicate: The message is sent twice with the same contents. 5. Removing the message: The message is lost and thus not received. The error detection and handling, and the contents of a message in the serial communic- ation link in this thesis is discussed in more detail later, in Subchapter 3.2. 29 2.4 SoC-FPGAs As the fault injector device in this thesis, a field-programmable gate-array (FPGA) is used. It is a circuit that is field-programmable or reconfigurable by the designer. An FPGA can be programmed by designing and specifying the logic with a hardware de- scription language (HDL). An FPGA consists of a high number of configurable logic blocks (CLBs) which usually implement a simple logical function or memory. When the FPGA is programmed, a configuration tool connects these multiple configurable logic cells of the FPGA together so that a circuit that implements the specified logic is cre- ated. Thus, an FPGA can implement a logic of an application-specific integrated circuit (ASIC) but has the advantage of being reconfigurable which increases available devel- opment time and reduces costs. (Shannon 2012: 127). Because an FPGA does not contain any logic itself, designing the logic must be done from the beginning. Thanks to the possibility to fully customise the logic, FPGAs are faster in signal processing than a CPU (central processing unit) but because of the lack of overhead such as predefined processor instructions and peripherals like memory units, the development is very time consuming. So, usually it is more reasonable in an application to use a microcontroller or a pro- cessor where some of the more complex calculations are done, and use an FPGA where data processing that requires low latency and speed is needed. To create this kind of a system, a stand-alone CPU and an FPGA alongside with other peripherals can be used. A more economically sound choice is to use a soft core, i.e. to implement the CPU logic on the FPGA itself – since any logic, even a CPU can be programmed on an FPGA. One alternative is to use a SoC FPGA. A SoC FPGA is a single integrated circuit that has a full processor architecture includ- ing a central processing unit, memory and inputs and outputs (I/O) and other peripherals such as networking interfaces on a single chip. The chip is relatively small, so it can be used in applications where circuit size is limited, and low power consumption is needed. This processor is integrated with an FPGA onto a single chip to utilise the best parts of the both. The advantage of this is that, unlike in a system consisting of a separate micro- 30 processor and an FPGA, the SoC FPGA is on a single silicon, decreasing the distance between the signals that are transmitted between the two, thus decreasing power con- sumption, response time and manufacturing costs. (Altera 2013: 1–4). 2.4.1 Advanced eXtensible Interface The Advanced eXtensible Interface (AXI) is an interface protocol defined in ARM (ad- vanced RISC machines) advanced microcontroller bus architecture (AMBA) specifica- tion. It is widely used protocol for developing applications for SoC-FPGAs. The AXI protocol defines the rules how the data is exchanged between intellectual property (IP) cores. (Xilinx 2012: 5). Thus, the AXI protocol can, for instance, be used to set up the communication between the programmable system (PS), i.e. the microcontroller and the programmable logic, the FPGA. In the use case of the thesis, the programmable system can receive data over net- work from the PC tool, process it, and write the processed data with AXI protocol to the fault injection logic block, which is an FPGA IP block. The communication with an AXI protocol starts with a handshake process during which the master of the AXI communication and the slave indicate that they both are ready for the data transmission. After a successful handshake, the data is transferred over the se- lected channel, depending what kind of action is desired. There are five different chan- nels: write address channel, write data channel, write response channel, read address channel and read data channel. All the channels have their own handshake channels, and the handshake is executed using the handshake channel of the corresponding action. For example, if the AXI master intends to read data from the AXI slave, the address is trans- ferred over the read address channel, and the read data is transferred to the master with the read data channel, as shown in Fig. 10. (Xilinx 2012: 5–6). 31 Figure 10. The read transaction uses the read address and the read data channels to read data (Xilinx 2012: 5). Figure 11 illustrates the channels and the data used during a write transaction over the AXI protocol. For example, when the master requests to write data, a handshake is done on the write address channel, after which the data, the address where the data write should occur, is sent by the master. After this, another handshake is executed on the write data channel, and the master sends the actual data which should be written to the address. Finally, the slave sends an acknowledgement for successful data write by using the write response channel. Figure 11. Write channels (write address, write data & write response) in the AXI in- terface and the flow during the write transaction (Xilinx 2012: 6). 32 2.4.2 Internet protocol and user datagram protocol The fault injector device which is implemented in the thesis, will communicate with the PC tool over Ethernet – this should not be confused with the serial communication link where the errors are generated. Thus, implementing the communication requires know- ledge also on the protocols that are used to transfer data over Ethernet. On the transport layer, the user datagram protocol (UDP) is used together with the internet protocol (IP) underneath it on the internet layer and, as stated, Ethernet on the link layer. An alternat- ive for UDP could be transmission control protocol (TCP). The internet protocol is the protocol that takes responsibility to route the transferred data between the networks. The actual data is in a packet of a higher-level protocol (such as UDP or TCP) and when sending such a packet over the Internet Protocol, it is encapsulated in a IP packet. The IP packet adds additional headers into the data, such as IP address which is used to route the package into the correct network and to the correct recipient. Furthermore, the IP packet is wrapped into a frame in the link layer protocol (e.g. Ethernet) and this data is passed into the transfer medium. An example with the data encapsulation hierarchy (with UDP/IP) can be seen in Fig. 12. (RFC 791: 1–11). Figure 12. Data encapsulation with UDP and IP. Also, the Internet Protocol takes responsibility of fragmentation and reassembly of the data. This means that if the link layer below the IP layer has restricted length for the 33 frame it can transfer, the Internet Protocol will slice the data in smaller fragments to di- vide it into several frames. Upon the receiving, the receiver will then reassembly the fragmented data. (RFC 791: 7–8). With the UDP protocol, applications can send datagrams, i.e. data packets, over an IP network. As depicted in Fig. 13, a UDP datagram consists of a header field which has the destination address (IP address) for the datagram, the data length and optionally a source port and a checksum to validate the data. (RFC 768: 1). When the IP protocol is used, also a pseudo header is added before the UDP packet’s own header and it consists of the source and destination IP addresses as well as the pro- tocol identifier and the packet length. This pseudo header has no functional meaning; the IP addresses here are not used to route the data, but they are only a part of the check- sum calculation. (RFC 768: 2). Figure 13. UDP datagram with IP (version 4, IPv4) pseudo header in green, the UDP datagram header in blue and the actual information in red. The pseudo header takes up 3×4 bytes and the UDP header 2×4 bytes so the total size of the header is 20 bytes. In UDP, no connection between the transmitter and the receiver needs to be established to transfer data. Thus, without a connection, the application cannot make sure if the da- tagrams have been delivered successfully or that in which order they are received. This may restrict the use of UDP in some applications because lack of error handling, but it 34 also has advantages: unlike in the TCP, which would be an alternative protocol for com- munication over internet, there is no handshake process to establish the connection or a disconnecting process between the parties, but only direct delivery of the datagrams which decreases the delay time. Also, because of the lack of acknowledgement on successfully transmitted datagrams, the data that is not delivered, is just dropped out instead of requesting the retransmission of the data. This is advantageous in real-time applications such as audio or video streaming to decrease the number of interruptions in the data because of error correction and retransmission of the data. If the communication robustness in the application is critical, TCP can be used instead of UDP – or the application that is using UDP can im- plement own error handling or connection mechanisms in the application protocol over the UDP. The two transport layer protocols are compared in Table 3. In the comparison, it can be seen that UDP excels over TCP with high speed and low load but is less robust. 35 Table 3. Comparison of the basic attributes of the UDP and TCP protocols. (Wiki- books 2019). Feature UDP TCP Description Simple high speed low func- tionality wrapper that inter- face applications to the net- work layer and does little else. Full-featured protocol that al- lows applications to send data reliably without worrying about network layer issues. Connection setup Connectionless data is sent without setup. Connection-oriented; connection must be established prior to transmission. Data interface to the application Message base -based; data is sent in discrete packages by the application. Stream-based; data is sent by the application with no particular structure. Reliability and ac- knowledgements Unreliable best-effort delivery without acknowledgements. Reliable delivery of message, all data is acknowledged. Retransmissions Not performed. Applications must detect lost data and re- transmit if needed. Delivery of all data is managed, and lost data is retransmitted automatically. Features provided to manage flow of data None Flow control using sliding win- dows, window size adjustment heuristics and congestion avoid- ance algorithms. Overhead Very low Low, but higher than UDP Transmission speed Very high High, but not as high as UDP Data quantity suitab- ility From small to moderate amounts of data. From small to very large amounts of data. 2.5 Robot Framework test automation framework Robot Framework is a generic automation framework for testing written in Python pro- gramming language. In this thesis, Robot Framework is utilised to generate test func- tionality and to run tests. With Robot Framework, first of all, the user can use a tabular based syntax to create a test case and start its execution from the command line. 36 Secondly, Robot Framework is extendable; the user can create own keywords – which effectively behave like functions – each of which performs a user-defined action on the system under test. Thirdly, the user gets reports from a test that is run by Robot Frame- work. (Robot Framework 2019). As stated, Robot Framework is modular, as it provides the framework for the test data syntax to pass the test data and the application programmer interface (API) to imple- ment the test actions for the system under test, the communication link, as shown in Fig. 14. So, the user can extend the framework by creating own keywords (functions); there are some built-in keywords exported by the Robot Framework itself but Python and Java programming languages are used to create test libraries, i.e. a unit containing multiple keywords. The keywords contain implementations that execute actions that test or valid- ate the system under test either directly or by using a test tool. These keywords can then be called by a test case file that uses and has the access to the defined test library. (Ro- bot Framework 2019). Figure 14. Robot Framework provides the test data syntax and the application interface (API) to create tests for the system under test. (Robot Framework 2019). Robot Framework provides also reporting and logging tools to gather data about the test. While running a test, the user gets informed of the test execution by log messages in the terminal. After the test is completed, an XML (eXtensible Markup Language) and 37 an HTML (Hypertext Markup Language) files are generated which the user can view on a web browser to easily take a look how the system under test behaves. (Robot Frame- work 2019). Following is an example of a Robot Framework test file, which tests a calculator applic- ation. Here, the example uses a custom made test library called CalculatorLibrary. The test suite contains three test cases called Additions, Subtractions, and Calculation er- rors, the first two of which have two steps and the last test case, Calculation errors, only one test step. These test cases are run in order when the test suite is executed. As shown in Algorithm 1, a user can define keywords not only in the test library with Python or Java but also in the test file, under the “*** Keywords ***” notation (here, “Calculate” and “Calculation should fail”). The user can define the arguments that can be passed for the keywords and then the test steps that are executed when they are called. As can be seen, the keyword Calculate takes two arguments and calls the Push buttons keyword from the CalculatorLibrary which operates the action with the calcu- lator application and also the Result should be keyword which validates the result. *** Settings *** Library CalculatorLibrary *** Test Cases *** Additions Calculate 12 + 2 + 2 16 Calculate 2 + -3 -1 Subtractions Calculate 12 - 2 - 2 8 Calculate 2 - -3 5 Calculation errors Calculation should fail 1 / 0 Division by zero. *** Keywords *** Calculate [Arguments] ${expression} ${expected} Push buttons C${expression}= Result should be ${expected} Calculation should fail [Arguments] ${expression} ${expected} ${error} = Should fail C${expression}= Should be equal ${expected} ${error} Algorithm 1. Example of a Robot Framework test sequence, containing a test case and custom keywords. 38 The test library, CalculatorLibrary, can be implemented with Python to actually interact with the system under test, the calculator application, and it is implemented as a Python class, as shown in Algorithm 2. When a test is run by Robot Framework, it searches for the source files of the libraries that are defined in the test sequence, and calls the Python (or Java) function, when that step occurs in the test sequence. from calculator import Calculator, CalculationError class CalculatorLibrary(object): def __init__(self): self._calc = Calculator() self._result = '' def push_button(self, button): self._result = self._calc.push(button) def push_buttons(self, buttons): for button in buttons.replace(' ',''): self.push_button(button) def result_should_be(self, expected): if self._result != expected: raise AssertionError('%s != %s' % (self._result, ex- pected)) def should_fail(self, expression): try: self.push_buttons(expression) except CalculationError, err: return str(err) else: raise AssertionError("'%s' should have failed" % ex- pression) Algorithm 2. An example of implementing a custom library which provides functions that can be used in Robot Framework tests. If the test sequence (in Algorithm 1) and the test library (Algorithm 2) are located in the current working directory, and Python and Robot Framework are installed, the previ- ously given calculator test example could be run by executing a Python command python -m robot calculator_test.robot in the command line, where “-m robot” as a parameter means that a Python module called “robot” (Robot Framework) is used, and “calculator_test.robot” is the file name of the test sequence. The framework runs the test by using the keywords in the test file and in the test library, and after the execution, test result is printed on the console and 39 the reports from the test are generated for more in-detail inspection. As told, it is also possible to use Java programming language to implement the test libraries. However, because in this thesis, the test library is implemented in Python, any examples of how test libraries can be written and tests can be run with Java, are not described here. In the previously described style, such Python keywords (functions) will be designed and implemented in the following chapters for the error generator system to inject faults. Of course, instead of pushing buttons of a calculator, they will send fault injec- tion commands to the SoC-FPGA fault injector which will use the fault injection FPGA block to inject the faults. Now that all the relevant theory for the error generator system is studied, the details of the system are discussed. After this, the design and the implementation of the system can begin. 40 3 ERROR GENERATOR SYSTEM Figure 15 depicts the proposed structure of the error generator system. This chapter de- scribes each part of the error generator system in more detail in the subchapters. Figure 15. The parts in the proposed solution of the error generator system. The figure is already introduced in Subchapter 1.2 about the objective of the thesis. The objective contains creating the components of the system that are inside the red dashed-line rectangle in the figure, thus excluding the fault injection lo- gic block and the communication link or the nodes. The focus of the description is more on those parts of the error generator system that are in the objective of the thesis. As a recall, these include the parts inside the red dashed- line rectangle in Figure 15: 1. Robot Framework test library (RF test library), which will provide the logic that makes possible to create and run fault injection tests. Manual test templates are also provided. They can be used as reference when running the tests. 2. Communication between the test library on a personal computer (PC) and the SoC-FPGA fault injector. The communication is carried out over Ethernet with the UDP/IP protocol and is shown in the Figure 15 as “Ethernet cable”. 41 3. Communication between the programmable system and the fault injection logic. The programmable system will receive injection requests from the Robot Frame- work test library over Ethernet and pass them to the injection logic by using the provided AXI interface. Possible responses from the fault injector should also be handled properly. The other parts of the system (high-speed communication, the communicating nodes and especially the injection logic block) are also described shortly but they are not im- plemented in this thesis. They are delivered either during the thesis or later by Danfoss. These parts are not needed for the implementation phase because the previously de- scribed parts can be implemented and tested separately. Creating the software for the programmable system depends on the fault injection logic IP (intellectual property) block. However, this does not block the development process. It can be implemented even if the block were not delivered as long as the description for the AXI interface is provided. This way the communication over the AXI interface between the programmable system and the injection logic FPGA block can be designed and implemented. 3.1 Danfoss test automation system The fault injector will be a part of the Danfoss test automation system as shown in Fig. 16. With the error generator system, it can be validated that the communication between the nodes works expectedly. All the three parts designed and implemented (the SoC-FPGA fault injector, the Robot Framework test library and the communication between them) are used in the test auto- mation system. Firstly, the Robot Framework test library is downloaded into a server machine which will automatically execute tests. Secondly, as can be seen in Fig. 16, the SoC-FPGA fault injector will be connected into the test automation server (instead of a personal computer) when it is integrated in the test automation system. Finally, the im- plemented Ethernet connection will be used in the test automation system to enable communication between the RF test library and the SoC-FPGA fault injector. 42 Figure 16. The schematic view of how the test automation system looks like. The error generator system designed (RF test library, Ethernet communication and SoC-FPGA fault injector) in this thesis will be a part of it. In the test automation system, the tests are run and the results are collected automatic- ally. Automatic tests with comparison with manual tests are discussed more in Subchapters 3.4.2 and 3.4.3. 3.2 High-speed communication link A serial communication link will be connected to the fault injector for fault injection testing. The parties in the communication link are FPGA IP blocks inside the nodes, e.g. option boards of a frequency converter. The nodes transmit real-time data and configur- ation data between each other over this serial communication link. However, the con- tents of the data are irrelevant from the viewpoint of the thesis. The high-speed communication link uses a proprietary communication protocol de- veloped in Danfoss. It uses a 8b/10b encoding which is an extended line coding. It maps 8 bit words into 10 bit words which decreases the noise-to-signal ratio and reduces the DC offset because, in average, the number of 0 and 1 bits in the data are in balance after additional bits are inserted. Furthermore, with the increased number of bits, there are enough state transitions for synchronisation without significant increase in bandwidth. (Widmer & Franaszek 1983: 450). 43 The packet, or the message, in the protocol consists of a message target address, data and a checksum. With the target address, the receiving IP block can route the message internally to the correct recipient within the block. The data integrity is validated with a CRC checksum and the most critical messages also include a sequence number. In the protocol, the error management relies on the error detection by the CRC. The re- ceiver drops a message if it is detected invalid by the CRC block. However, the receiver does respond with neither a positive or a negative acknowledgement message upon re- ceiving. This is because the transfer speed is extremely important in the communication. To increase the speed and to limit the bandwidth usage of the communication link, auto- matic repeat requests (ACK messages) are not sent, or invalid messages are not retrans- mitted. Nevertheless, to be robust, the transmitter should send a critical messages twice. 3.3 SoC-FPGA fault injector The fault injector itself is a SoC-FPGA, as shortly described in the objectives of the thesis. The SoC-FPGA fault injector will contain the programmable system (a pro- cessor) and the injection logic IP block (FPGA) developed by Danfoss. The communic- ation link, where the faults will eventually be injected, is connected to the latter, the in- jection logic block. 3.3.1 Fault injection logic of the SoC-FPGA fault injector The fault injector logic block is an FPGA module that will be developed by Danfoss and will be integrated into the SoC-FPGA fault injector. It takes the serial data in, injects a fault into the correct package depending on the values passed from the programmable system over the AXI interface, and passes either unmodified or modified serial data out. The injection logic will be connected to the programmable system as an IP block by us- ing the provided AXI interface. The AXI interface of the injector logic block will (at least) contain the following data input: the fault type, the target address and the number of repetitions, i.e. how many times the error will be injected. With the target address in- put, the injection logic finds a message which contains the given target address and in- jects the fault to the message. The fault type is one of the following: 44 1. CRC fault: The injection logic flips bits in the message data so that the CRC cal- culation is violated. The CRC field is kept as is. 2. Change target address: The injection logic changes the target address of the mes- sage. The receiving node would route the message according to the address to a wrong block inside the node. 3. Increasing/decreasing the length of the message: The injector removes or adds data in the message. 4. Duplicate: The message is sent twice with same contents. 5. Removing the message: The message is lost by the injection logic and thus not received. The nodes should detect the errors (e.g. by using the CRC sum) in the messages and re- act accordingly. The response to an invalid message depends on i.a. how critical the message and the fault are and thus cannot be generalised. In extreme cases, precaution- ary measures, i.e. motor shutdown, may be carried out by the receiver if an error is de- tected. The injection logic provides data output for the programmable system, too. The injec- tion logic acknowledges a successful fault injection to the communication link by passing an acknowledgement over the AXI interface to the programmable system. The acknowledgement contains the type of the fault that was injected. The programmable system should handle these responses properly and pass the acknowledgement forward to the Robot Framework test library. However, the injection logic can only return data about the fault injection process itself. It can give information if the fault could not be injected but it does not know whether the receiving node processed the fault correctly. This should be managed by the enclos- ing framework, the test automation system (see Subchapter 3.4.3 Automatic tests). 3.3.2 Communication between the fault injector and the test library The fault injector is connected to Robot Framework test library on a PC over Ethernet. The communication is built on UDP/IP. Thus, the processing system of the fault injector must be able to handle UDP datagrams and parse the request of the datagram sent by the 45 Robot Framework test library. Furthermore, the fault injector system has to be able to send data in the other direction, back to the PC, too. Even though TCP/IP alternative would have been a more robust alternative for commu- nication between the fault injector and the test library, UDP was selected because it is more lightweight and faster as it was discussed and compared in Subchapter 2.4.2. Furthermore, and more importantly, the SoC-FPGA board of the fault injector will be used for other operations (outside this thesis) where UDP will be used. So for better maintainability and extensibility, using UDP was included in the specifications so that only one protocol would be used in the SoC-FPGA fault injector on the transport layer. As a conclusion, UDP was selected mainly for the latter reason, and no additional com- parison is made in this thesis on the speed, the bandwidth-usage and the reliability between TCP and UDP. In this thesis, a simple application protocol is built on top of the UDP to send data between the RF test library and the fault injector. Without it, the contents of datagrams sent by UDP are meaningless themselves. The protocol should describe the format in which the requests to the fault injector and the responses from it are represented in a UDP datagram. If the fault injector needs setup data, the communication protocol should also define the format for it. 3.4 Test library with Robot Framework 3.4.1 Robot Framework keywords for fault injection Robot Framework keywords are written in Python programming language as Python functions. There is one keyword for each of the five fault types described in Subchapter 3.3.1. Each keyword takes also a repeat number and a target address as their paramet- ers, so that the number of injected faults and the target address of the message can be configured. The flow of the keyword is shown in Fig. 17. 46 Figure 17. A sequence diagram describing how the data flows from a Robot Frame- work keyword (RF keyword) to the fault injection logic and eventually into the communication link as a generated error. All the keywords have somewhat same functionality. When the keyword is executed, a UDP datagram containing the fault type, repeat number and address is generated and sent over Ethernet to the fault injector. Before sending the data, the keywords wrap the information in a datagram that has a predefined format so that the injector can read and understand the data. This format will be specified in Subchapter 4.2. After the datagram is sent, each keyword should wait for a UDP response datagram from the injector. The fault injector validates the request that is sent. It also gets the re- sponse from the fault injection logic that the fault was injected successfully. If the re- sponse was positive, the injector sends a UDP datagram containing a positive acknow- ledgement, otherwise a negative acknowledgement. Thus, if the keyword receives a negative acknowledgement or does not receive any acknowledgement within a defined time period, it should mark the keyword execution as failed. This is done by raising an exception; the built-in features of Robot Framework then stop the test execution and mark the test as failed with a message that is given by the keyword. 47 In addition to the five keywords which are used to inject faults, there will be additional keywords to execute setup and utilisation functions on the fault injector. Setup keywords can include changing IP addresses and ports of the fault injector or fetching version information. 3.4.2 Manual (targeted) tests In manual tests in general, the person running a test, a tester, manually runs a test by fol- lowing a predefined test plan. Furthermore, the tester reads manually the output of the test program and validates the output to determine if the test was successful or not. In the fault injection testing, a Robot Framework test file is started manually by the tester on a PC. The test sequence in the test file contains a list of fault-injection keywords that are to be executed. Then, after the test has finished, statistics are collec- ted from nodes of the communication link, and they are manually read to validate that the fault detection and management behaviour of the nodes worked expectedly. Also, during the test execution, the tester should monitor log messages of Robot Framework to see whether the actual fault injection operations are executed successfully. With targeted test, the tester can specify which faults should be injected. This can be useful, when only certain fault types or a specific target address needs to be tested. However, unlike in automatic tests, the tester needs to be present during the test and has to read the test output manually. Also, unless the test results are in a clear format, the tester must know how to interpret the read test output correctly to define the result of the test. 3.4.3 Automatic tests With automatic tests, a test process is done completely automatically. Tests execution are started and executed automatically either by a trigger or at a specific point of time, e.g. once a day. The results of the tests are read automatically and stored in a human- readable format. Usually, if the automatic tests are run periodically, the operator of the automatic test system is notified in case the test were not completed successfully, so that proper actions can be carried out. 48 The fault injection tests will be executed in the test automation system. The test automa- tion system will consist of a server machine, which has a script that will run a Robot Framework test sequence automatically. The sequence contains all the fault injection tests so that all faults will be injected and tested automatically. After the test has run, the test automation system is capable of collecting the statistical data from the nodes to de- tect, whether the faults were detected and handled correctly in the receiver. Collecting statistics about the communication and analysing them is done outside of the error gen- erator system and thus is outside the scope of the thesis and not discussed here in detail. The advantage is that the tests could be automatically run when a new software binary is downloaded into a node. Because a tester does not have to be present, tests with longer duration and a wide range of target addresses and repetition count can be automatically executed overnight and results can be read next day. However, running the automatic tests here requires that the development of the test automation system – which this error generator system will eventually be part of – is finished. Because that development pro- cess is not the objective of the thesis, it is possible that results of automatic tests cannot be included in this report. The components of the error generator system are now described. As a recall, the error generator system consists of 1. Robot Framework test library, which will be used to create and run tests and which communicates with the SoC-FPGA fault injector over Ethernet, 2. SoC-FPGA fault injector, which has a processing system (a processor) and a fault injection FPGA block, latter of which is not created in this thesis, and 3. the serial communication link and the nodes, e.g. option boards, which commu- nicate over the link, and which are not created here but provided for the testing phase. Now that the system is described, it is designed and implemented for those parts that are included in the objective. Those include the RF test library and the SoC-FPGA fault in- jector and the UDP/IP communication between them. 49 4 DESIGNING THE ERROR GENERATOR SYSTEM This chapter describes how the error generator system is designed. The design process proceeds hierarchically, starting from an abstract structure description of the system, eventually ending into designing the detailed flow of a single algorithm. Design tools such as component diagrams, class graphs and pseudo code are utilised in the design process. The design starts with defining the top-level components and their responsibilities for the error generator system. From the requirements in the Subchapter 1.2 describing the objectives of the thesis and the description of the system in the previous chapter, three main components of the design could be identified easily. The initial component hier- archy is shown in Fig. 18. All these three main components are then designed individu- ally and the design is described in the following subchapters. Figure 18. The objective consists of designing the test library, (the programmable sys- tem of) the fault injector and the UDP/IP communication between them. As stated, the fault injection logic block (the dashed rectangle on the right) is not in the objective, but will be used in the ready system. The UDP/IP communication is the backbone of the whole system which is why it is de- signed first. It defines the message format and the supported messages for the commu- nication to make the communication between the test library and the fault injector pos- sible. To increase the robustness, an error handling mechanism can be designed, too. 50 After the UDP/IP communication is designed, the test library and fault injector design process can begin. The Robot Framework test library has a UDP handler block, which supports the previ- ously mentioned UDP/IP communication and its message formats. It is responsible of sending the messages and receiving them. Eventually, the messages that are handled by the UDP handler, are sent and received by the Robot Framework keywords. They are split into two sets; the fault injection keywords execute a specific fault injection test and the setup and helper keywords change settings in the fault injector or provide additional functionality. These all are contained in a Robot Framework Test Library, implemented as classes and functions in the Python programming language. The test library also has example test sequences implemented as Robot Framework test files. The test files are templates to show how the fault injection tests and setup func- tions in the Robot Framework test library can be called in a Robot Framework test file. This template can then be used to create and execute manual tests in the testing phase. Like the RF test library, the programmable system of the fault injector contains a UDP handler module which handles the incoming and outgoing UDP datagrams. The UDP handler of the PS then parses the request and calls the AXI handler. The AXI handler sends the fault injection request to the fault injection logic block over the AXI interface, gets the response and returns it back to the UDP handler. 4.1 Design of the UDP/IP communication Here, the design of the communication with UDP between the RF test library and the SoC-FPGA fault injector is given. First, to ensure reliable UDP communication, a method to increase robustness is selected and fine-tuned. Then, the actual format of a datagram is designed by first recalling the requirements and then defining the format. 4.1.1 UDP communication data flow As already stated, the UDP protocol is a connectionless protocol. This means that to send a datagram, a communication between the sender and the receiver does not need to 51 be established. This increases speed but the sender cannot be sure whether the receiver has got the datagram. Our protocol over the UDP communication must address this is- sue somehow. A simple way to ensure a successful transmission is to use acknowledgements. This, however, would require that acknowledgement and the retransmission procedures are designed, which causes extra work. This would also use extra bandwidth and cause delay which decreases the advantage of using the UDP. Furthermore, and most import- antly, defining an acknowledgement procedure in our protocol would be reinventing the wheel since such mechanism is already implemented successfully in TCP. To understand the effects of the errors on the communication link, different error hand- ling mechanisms were studied in Subchapter 2.2.3. This information, however, turns out to be useful when designing the UDP communication. For blind transmission, it was given that “to reduce the number of lost packets, the transmitter can send a packet mul- tiple times and ‘hope’ that at least one of them will receive the destination”. So in our UDP protocol, all the messages are sent three times so that at least one data - gram will be received properly. However, to avoid duplicates, the message must contain an increasing message identifier number, so that only one message is handled and the remaining messages with the same identifier are ignored. This is to avoid accidentally generating a same fault multiple times in the fault injection logic block; the error gener- ator system could provide ambiguous results in testing, when a successful transmission of a package was expected in the link, but an error occurs unexpectedly because of a du- plicate fault injection command processed by the SoC-FPGA fault injector. 4.1.2 UDP datagram structure All the data is sent as characters in the datagram. This way, the implementation can be made faster when the datagram contents are in a more human-readable format in strings of characters instead of plain bytes. The most of the datagrams are created in the Robot Framework test library on Python where manipulating character sequences (strings) is fast and easy, though manipulating bytes is possible, too. Sending the data as characters 52 significantly increases the datagram size compared to sending as “raw” bytes, so they must be kept as short as possible. However, recalling from Subchapter 2.4.2, Figure 13, that the headers of a UDP data- gram already take up 20 bytes of the total size. Using the binary format does compress the size of the payload but the relative decrease in the total size of the datagram is not significant, if the length of the data in the string format would be around the length of the header, as it is in this use case. As previously stated in Subchapter 4.1.1, the datagram should contain an identifier field to avoid duplicates. This is done by starting the datagram with a one-character (byte) field containing a number from 0 to 9 (the message identifier field). The message should start with this identifier and the receiver should drop a datagram with an identifier as the same as that of the previous datagram. No action is needed when the message identifier shifts by more than one value between the consecutive datagrams. To identify, what kind of message is sent by the datagram, the datagram type field is ap- pended to the datagram after the message identifier separated by a space character. Four characters are reserved for this field for extensibility purposes. The message type can be for example a request to inject a fault or it can be an acknowledgement from the fault injector that the fault injection was successful. These types are discussed more in Subchapter 4.1.3. Furthermore, some messages need additional information, i.e. parameters. These are ad- ded to the end of the datagram, again after a space character separating them from the previous field. If more than one parameter is sent, they are separated by a space. The datagram format is now complete. The full datagram format is shown in Table 4. As can be seen, the minimum length of the datagram is six (6) characters if the message type does not require parameters. Thus, if the receiver receives a datagram shorter than that, it can safely ignore it. The maximum length of the datagram is defined later, when the message types are defined, because the maximum length of a datagram depends on the maximum length of the combined parameters at the end of the datagram. 53 Table 4. The datagram format with the field names and the length for each field. Field Message identifier (0–9) Space Message Type Space Parameters Length (chars) 1 1 4 1 1–… Required yes yes yes no no 4.1.3 Message types for a datagram After the datagram format is designed and defined, the contents of a datagram are defined. These include defining the message types and the possible parameters of a mes- sage. The fault injection keywords of the test library send the fault injection requests to the fault injector. Thus, because there are five possible types of a fault, each type for a fault injection must have its own message type. The fault injection needs also the target ad- dress and the number of repeats to be successful, as stated in the description of the fault injection logic, so these two values must be sent as the parameters of a fault injection datagram. Again, the type identifier is defined as a string of characters to make it easier to develop the communication handling in the test library and in the SoC-FPGA, and also, when later testing the UDP communication, to make debugging the received and sent datagrams easier. Furthermore, from the setup and helper keywords, the following data can be sent to the fault injector: set IP address, set UDP port, set communication link type and get the ver- sion information of the fault injection logic IP block. Here, setting the IP address and the port refers to the address and port of the PC running the Robot Framework, so that the fault injection logic knows in which address send the data back. Naturally, these two take an IP address or a port as their parameter. The message type to set the communica- tion link type has one numeric parameter that describes the communication link type. Finally, there should be a possibility to send a “test started” message from the RF test library to the SoC-FPGA fault injector at the beginning of a test. With this, the fault in- jector can reset its message identifier number, so that if the very first message of the test sent from the Robot Framework is the same as that of the last test, the datagram is not accidentally ignored as a duplicate. 54 All the possible messages that can be sent from the RF test library to the SoC-FPGA fault injector are defined. The message types with their four-character identifiers and the possible parameters are listed in Table 5. Table 5. The proposed message types and parameters from the RF test library to the SoC-FPGA fault injector. Message description Message type string Parameters (Inject) CRC fault fcrc target address, repeats Address change fault fadr target address, repeats Length change fault flen target address, repeats Duplicate message fault fdup target address, repeats Drop message fault fdrp target address, repeats Set IP address sipa IP address Set UDP port spor UDP port Set communication link type sclt communication link type Get injection logic version gilv Start test sequence strt Also, the fault injector sends data via UDP back to the Robot Framework test library. These messages follow the previously defined datagram format. The fault injector re- turns the positive or negative acknowledgement about the injected fault, and the fault in- jection logic block version, when requested. The message types with their parameters are shown in Table 6. To keep the consistency with the four-character message type definition, an underscore is prefixed at the beginning of the three-letter message type identifiers. 55 Table 6. The proposed message types and parameters from the fault injector to the RF test library. Message description Message type string Parameters Positive acknowledgement _ack Negative acknowledgement _nak Injection logic version in- formation _ilv injection logic version The UDP handler modules of the RF test library and the programmable system on the SoC-FPGA fault injector should follow these datagram format definitions. For example, when a request to inject a CRC fault 5 times to address “12345678” is sent from the RF test library, a following datagram will be sent three times 0 fcrc 12345678 5 and if the fault was injected successfully, the fault injector would respond with 0 _ack 4.2 Design of the Robot Framework test library The Robot Framework test library consists of the test libraries and a template test se- quence or sequences. Furthermore, the test library will have fault injection keywords (Robot Framework’s “functions”) and set-up or configuration keywords. A UDP hand- ler will be included in this Robot Framework test library to provide communication, but it will a separate Python module or class and not directly in the test library. The test sequences should contain templates to show how the keywords are used when running a test with Robot Framework. There should be templates and examples how to create and run a fault injection test and how to use the setup functions. These test design templates can be designed either here on in the test preparation phase. This hierarchy and the relationships the parts is depicted in Figure 19. 56 Figure 19. Components of the test library. The test library utilises the UDP handler component to execute the keywords. The test templates are test sequences with examples how to call the keywords. 4.2.1 Design of the fault injection keywords Effectively, all the fault injection keywords are the same meaning that the body of a Py- thon function is almost identical for all the functions. Only the type of the fault injection changes, which, as described in Subchapter 4.1.2, means that the message type is differ- ent. Thus, this nuance can be ignored when designing all the keywords at once. The flow diagram describing how the process in a fault injection keyword proceeds, is depicted in Fig. 20. The elements that are on the right side of the dashed vertical line represent actions of which logic is encapsulated in the UDP handler component, and, thus, are designed later. 57 Figure 20. The flow diagram of a fault injection keyword. The steps located on the right side of the dashed line represent the actions that happen in the UDP handler which will be designed later. Firstly, each fault injection keyword takes a target address and the number of repeats as their parameters. First, the parameters should be validated when executing a keyword. Currently, no requirements are specified for the target address so any non-empty value should be accepted. Nevertheless, the number of repeats should have a numeric value that is greater than 0. Maximum value is not specified – unless more specifications are provided. After the parameter validation, the UDP handler is utilised, as can be seen in Fig. 20. The parameters, and the fault injection type which is included in the keyword itself, are 58 passed to the UDP handler to create a datagram with correct format. Then, this datagram is sent by the UDP handler, which takes the responsibility to send the datagram as it is designed. All the errors in this phase should be considered fatal and lead to stopping the execution of the keyword. The keyword waits for the acknowledgement after it has sent the fault injection request. Yet again, the UDP handler receives the datagram and returns it to the keyword. If the datagram was received within a configurable timeout, its contents are read by the UDP handler. If the response is a positive acknowledgement, the error injection was success- ful and this is output to the user. However, if anything else is received, the injection is assumed to be failed. In case of fail (timeout or a non-positive acknowledgement), an error should be thrown so that the execution of the whole test case will fail. 4.2.2 Design of the setup keywords As already shown in Figure 18 and