1.1 This document offers guidance on the reporting
of results of e-navigation testbeds.
2
Benefits and scope of the
guidelines
2.1 Harmonization of the reporting of results
from testbeds will allow the results of e-navigation solutions being
tested to be shared and compared effectively. Harmonization also allows
future meta-analysesfootnote of specific
aspects. Different organizations can recreate trials both to verify
results and refine various factors within the trials, in order to
further develop the concepts being trialled.
2.2 This guideline includes the following:
3.1 During the development of e-navigation, a
growing number of testbeds have been evaluated. Consequently, NAV
58 agreed to the development of Guidelines on Harmonization
of testbed reporting.
4.1 A testbed (also commonly spelled as "test
bed" in research publications) is a platform for trialling development
projects. Testbeds generally involve rigorous, transparent and replicable
testing of, for example, scientific theories, computational tools
and new technologies.
4.2 E-navigation testbeds allow for early detection
of new system functionality, operational usability, areas of enhancements
and identification of weaknesses. Ideally, testbeds should be linked
to human-centred design processes to ensure any operational usability
issues are detected early. Testbeds should not, necessarily, be limited
or restricted by current or planned architecture, data structures
or existing procedures. Considerations when planning a testbed are
given in annex 1.
4.3 Ideally, testbeds should be conducted in a
controlled environment so that they do not adversely affect real-life
situations, existing services and maritime safety. Conclusions can
be drawn for many aspects of testbeds such as functionality, usability,
feasibility and risk. As e-navigation evolves from concept to operational
reality, the importance of testbeds will continue to grow.
4.4 There are testbeds that, while being not directly
identified as e-navigation testbeds, are nevertheless relevant to
e-navigation. The reporting of results from such testbeds is encouraged.
5
Harmonization of reporting
of testbed results
5.1 As a number of testbeds are established, it
is important that the results of testbeds are shared, as there could
be outcomes and lessons learnt that will be useful to the maritime
community. In order to do this and to allow for ready comparison of
the relevant elements of testbed results, reporting of the results
of testing of e-navigation solutions, systems and services should
be harmonized.
6.1 For testbed results to be useful to other
parties, tests/simulations/trials should ideally have scientific rigour
for set-up, collection of data, analysis, etc. Additionally:
-
.1 the results presented should be objective;
-
.2 trials should be reproducible;
-
.3 data gathered should be statistically sound
and meet generally accepted "scientific standards"; and
-
.4 testbed results should be presented in acceptable
scientific formats (e.g. they should be suitable for publication in
a peer-reviewed publication).
6.2 A framework, by way of a template for reporting
has been developed (see annex 2) that addresses the presentation of
results. This should be taken into account when reporting results
of testbeds related to e-navigation to the Organization.