1 Introduction to Users and Experiences

ERIGrid integrates and enhances the necessary research services for analysing, validating and testing smart grid configurations. System level support and education for industrial and academic researchers is provided as well to foster future innovation. To this end, some internal and external programs have been defined to use the available research infrastructures provided by ERIGrid. From 2016 to 2020, ERIGrid enabled more than 100 research projects, in which most of the research teams used the Holistic Test Description (HTD) approach [1] for documentation of the test procedures and results as an effective system-level validation approach. This pool of HTD experiences created a valuable potential to collect feedback, validate the methodology and therefore presented a chance to refine the methodology. In this chapter, the application areas of the HTD methodology are either used internally in ERIGrid or adopted for external usages as discussed. The evaluation of the system-level validation is addressed in and a summary of the HTD users’ experiences are explained. Also advantages and shortcomings of the HTD methodology based on the integrated feedback of the users are presented.

2 Application of System-Level Validation Approach in Projects

As mentioned in Chap. 2, the HTD method has been proposed in the ERIGrid project as a template-based documentation and system-level validation approach. To have a better picture of the HTD application experiences, they are clustered based on the System under Test (SuT) and Domains under Investigation (DuI) aspects. As shown in Fig. 1a the SuT of the projects has been categorized in six main clusters; namely: (i) Device-level testing, (ii) smart building, (iii) microgrid, (iv) low-voltage distribution network, (v) medium-voltage distribution network, and (vi) Supervisory Control and Data Acquisition (SCADA) system. Most of the HTD users chose the microgrid as the SuT. Device-level testing, such as evaluating the performance of voltage controller, inverter and transformer, is the second most popular category.

Fig. 1
figure 1

Analysed a Systems under Test (SuT) and b Domains under Investigation (DuI)

Full size image

Figure 1b represents the different DuI of the HTD users, which are categorized in four main domains; namely: (i) electric power systems, (ii)control systems, (iii) communications, and (iv)electricity markets. The total percentage of DuI percentage exceeds from 100% due to the domains overlap in the projects. So far, all the users studied power systems as the main domain with different focuses on control, communication and market perspectives.

3 Evaluation of Representative Test Cases

The application of the HTD which is the proposed system-level validation approach in the ERIGrid project, is analysed for two different test cases involving a single or several joint infrastructures. The results of this analysis are presented in this section.

Single research infrastructure test case: A converter controller testing has been validated following a complete testing chain: (i) pure simulation, (ii) Controller Hardware-in-the-Loop (CHIL), and (iii) Power Hardware-in-the-Loop (PHIL). The objective of this test case is to accomplish several experiments aiming at evaluating the impact of the control and the parameters of the controller in the performance. For that, the different droop controls (P-f and Q-V) have been characterized and the results are compared. The test cases have been formulated based on three main Purpose of Investigations (PoIs):

  • Characterization of the converter controller influence of the system performance (i.e., fine tuning of models using results of hardware tests).

  • Validation of model exchange among RIs.

  • Validate improved control system performance.

This test case with several PoIs becomes a complex example of the use of the methodology. Therefore, the application of the HTD was done for one purpose of investigation considering several sub-PoIs. Due to the specific requirements of the tests to be performed, the design of an experiment realisation plan becomes challenging as the information provided by the labs is usually not so detailed to ensure replicability across different research infrastructures.

Several joint infrastructures test case: The second evaluated example was the test of a web service provided by one research infrastructure using data from another research infrastructure. Its aim is to validate a software available as a web service by using the measurements registered in one research infrastructure and sent to the cloud using a certain protocol, i.e., Common Information Model (CIM) [2, 3]. This test case has two PoIs related to the validation of:

  • A state estimator web service.

  • A virtual research infrastructure.

From the formal point of view, the application of this example is somehow incomplete. Moreover, evidences in different fields of the template may lead to misunderstanding on the whole procedure. One of the most outcomes and lesson learned from the application of this example is underlining the importance of defining common definitions and a clear guideline for the application of the methodology.

4 Evaluation of the Holistic Test Description Methodology

A questionnaire has been designed and handed out to the people employing the ERIGrid services associated with the holistic testing and validation procedure in the context of test cases in different projects. The goal of this questionnaire is to document issues and shortcomings of the services, in order to improve them and document the iterative development process. The following services are addressed by this questionnaire:

  • Holistic test case description: Templates and possibly accompanying guideline material for documentation of test case, test specifications and experiment specifications.

  • System configuration description: Filling forms for documentation of the test system configuration as part of the holistic test case description.

  • RI Database: Web-based platform for formal RI component specification.

The Data Specification Questionnaire is directed at all researchers who have employed parts of the HTD procedure. The goal is to point out the improvement potential of the procedure, especially in the context of data specifications.

4.1 Results of Work with ERIGrid Services Questionnaire

From the collected answers the majority of the TA users who used the holistic test case description service, used the system configuration description service, and only a few of them used the RI database service, which is the web-based platform for formal RI component specification.

The templates seem to be quite clear and convenient. Most of users did not encounter any problems with the templates. However, some users mentioned that there are some repetitions and some points could be summarized in fewer points. For instance, some users found it hard to distinguish between the System under Test in the Test Case description and the Specific Test System in the Test Specification description. Also, some users had difficulties in understanding the differences among the TC/TS/ES descriptions. The difference between the Test Criteria and the Target Metric was not described well. Regarding the templates, the users mentioned that they generally work fine for a Single-Test/Single-RI test case, but they become harder to use when multiple tests and/or RIs are introduced.

4.2 Results of Data Specification Questionnaire

All the users who participated in this questionnaire employed the Test Case Template, the Test Specification Template and the Experiment Specification Template of the HTD procedure, however one of the participants also employed the Qualification Strategy, the Experiment Realization Process (with RI-Database), the Design of Experiments Methodology and Statistical Analysis.

Concerning the “variable mapping”, in case of differences between the user documentation and the realized experiment, no major difference was observed. However, some users had to modify the systems initially planned to adapt to the specification of the used equipment.

Regarding the “lab integration”, it consisted of three steps; first, the interface of the new components with the lab, second, the real-time interaction among the labs, and, finally, the refinement of the lab set up based on previous experiments. For instance, one user had to deal with the interface of new components with the laboratory, while the previous project of that user was based on simulations. The problem was that those simulations were not directly exportable to the real laboratory, so they had to understand how the laboratory was structured and the control program was operated in order to obtain the correct results. They also had to change the initial measurement method to achieve greater precision in the data.

Speaking of how the result data was obtained from the experiment and stored by the users, some users stored the results in a Matlab file format in which case no data conversion was needed.

5 Advantages and Shortcomings of Holistic Validation Methodology

According to the questionnaire results, the majority of the RIs had not used any specific testing methodology before the definition of the HTD. Few partners refer to the Holistic Testing Procedure, developed also in ERIGrid, as a previous relative experience. Furthermore, it was mentioned that no clearly defined methodologies existed previously for Multi RI tests.

It was noticed that the HTD is relatively hard to use, especially for the first time, as it requires a thorough understanding of the various definitions and concepts used in the methodology. On the other hand, some users mentioned that certain templates that have been provided proved to be very useful, as there is also a guide that explains how to use them properly.

Based on the questionnaire feedback provided by the partners, the HTD methodology presents several benefits, which are evidently not only in comparisons to previous testing methodologies, but also in the implementations of single and multi-RI experiments.

5.1 Advantages of the Holistic Validation Methodology

While the majority of previously used testing methodologies to be compared to HTD consisted of custom methods and practices that were different for each individual RI, nevertheless HTD presents obvious strengths:

  • It provides a complete overview of the test, and by following the testing procedure provided by ERIGrid (using the guidelines), the users can have heightened awareness of the context during the test setup.

  • It helps to structure the planning of an experiment—providing the capacity to split complex test cases into individually achieved subtests and provides a means for the user to think about various critical aspects regarding the tests in a systematic manner, easing the identification of the parameters that need to be accounted for when designing holistic tests.

  • If performed properly, it can support documentation and communication, as the definition of basic concepts can contribute in improving multi-domain comprehension of complex validation activities and encourage interdisciplinary collaboration among teams.

Based on user’s feedback, in experiments that include the participation of a single RI, employing the HTD methodology can provide key benefits to several parts of the testing process:

  • By clarifying the Purpose of Investigations and Test and Experiment Specifications, it can lead to faster preparation of the TC, while the structured planning and execution of tests eases the communication process of test plans and results.

  • The effort that the method requires for the definition and the organization of the tests proves to be quite helpful when there is a need to implement the same experiments and compare the results between two different RIs.

  • In HIL/CHIL simulations, this method is mostly efficient when you have to develop a new component. What is more, the simulation and HIL experiments can extend the results of the pure hardware experiment.

Additionally, the utility of the HTD seems to be extended for tests that include the participation of more than one RI, in different physical locations. Let it be noted that any advantages present in single RI experiments also apply to the multi RI experiments, in addition to the following:

  • It allows transparent translation between the system under test and the test setup and clearly determines the boundaries of the test setup among different partners. This can also assist in clarifying and distributing the workload between different RIs, avoiding misunderstandings during the experiment and improving interdisciplinary collaboration among teams.

  • It leads to the standardization of test cases and the adoption of a common naming/signal exchange convention, improving the multi-domain understanding of complex validation activities.

  • It offers the possibility to simultaneously use hardware assets at multiple RI without the need for transport of equipment or personnel, thus minimizing investment and transportation costs.

  • Web services provided by a single RI (such as a state estimator) can be used by other RIs without any exchange of software.

  • It can extend the validation capabilities of RIs and create advanced testing environments which cannot be implemented in a single RI mainly due to hardware limitations. It also extends the validation experience between partners (complementarity) and joint developments.

5.2 Shortcomings of the Holistic Validation Methodology

While constituting a noteworthy evolution of previous testing methodologies in several aspects, the HTD methodology does present certain insufficiencies, which the questionnaire aimed to identify. The answers of the participants converged to the following major points:

  • The HTD methodology is still complex, long and the presence of several different terms and parameters can make it appear too “academic”.

  • Some of its concepts/terminologies in the templates can be difficult to understand and similarities in their definitions can lead to different interpretations and render the implementation of the methodology relatively cumbersome.

  • It is missing a process for the collection and reporting of (sub) test results.

  • It is in need of more support documentations and adjuvant tools, like a comprehensive user interface or pre-filled templates as examples.

6 Conclusion

This chapter presents a comprehensive evaluation of the system-level validation approach that was employed for the tests by the majority of the research teams and is known as the HTD method. Initially, a summary of the application of the method in two characteristic test case instances is presented, thus gauging its utility through examples and offering a first impression of its strengths and weaknesses. In order to perform a more thorough assessment of the method, based on the particular experiences of the HTD users, some questionnaires were designed and distributed, and the feedback was utilized to form a more structured evaluation.

Consequently, the results of two questionnaires are presented, one regarding the ERIGrid services and one concerning data specification. The former examines the quality of the services relevant to the holistic testing procedure, namely the holistic test case description, the system configuration description and the RIs database, while the latter searches opportunities for improvement in the fields of system configuration, definition and naming of variables and data exchange.

The chapter closes with the demonstration of the results of a questionnaire addressing the general advantages and shortcomings of the use of the HTD by individual RIs. The pros of the method in contrast to older methodologies are highlighted, and additional merits are described in experiments conducted both under single RI and multi RI status. The final section presents feedback related to the perceived imperfections and drawbacks of the method, which can evoke future improvements and can be used to determine goals for its subsequent development.