Draft Response to be sent to the EC challenging comments in the Technical Review Report

This page documents text to be used to structure our response for D14.1, following its recent updates to address comments raised during the 2nd Review Meeting and Report.


Dear Manuela,

Following our joint review of the Technical Review Report at our recent All Hands Meeting held in Greenwich, London, on 15 May 2013, we would like to respond to specific sections of the report where the project participants either want to raise objections to some of the conclusions drawn by the reviewers, or else provide additional supporting information to further define the position of the project participants.

WP14 Common Test Environments

Following internal review, and subsequent discussion at the recent All Hands Meeting we would like to respond to the following points made in the technical review report.

Scope of Test Environments

In the final paragraph of page 7 of the technical report, the reviewers make the comment that: "No tests were performed on the test environments delivered by previous projects, nor on the four indicated testing environments offered by APARSEN consortium partners, ....

We accept that we can provide more evidence of the testing that has taken place within the four indicated APARSEN test environments and have revised Deliverable D14.1 to include a new Section 5 that addresses this issue. We do however want to challenge the request made by the reviewer team for the project to go beyond the scope of the DOW for WP14 by asking us to provide evidence of tests performed on test environments delivered by previous projects. Within our DOW we state (5th para) that: "we aim to produce a collection of test environments from any APARSEN partners....., and later in the description of Task 1420 we state: "To facilitate this, partners will make their test environments, procedures, test data and software available to other partners within the constraints of the funding available within the work package." i.e. we limited ourselves to the test environments that participants within the consortium can offer.

We realised at the start of the project that there are real issues around sustainability of digital preservation services (i.e. one of the main topics of the project!) and that the constraints of this project's funding would prohibit us from trying to resurrect test environment software from previous projects. We thought we had made this clear within the DOW. To be clear, we are not adverse to reviewing and including tests from new and emerging test environments within the DP domain, as this would be of a direct benefit to the objectives of the VCoE, and to this end following the review meeting we have contacted other projects to initiate this process. The SCAPE project is one such project where we have opened a dialogue with Tomasz Parkola (PSNC) to identify how APARSEN can collaborate with SCAPE when their Test Environments become available for use in September, later this year.

We have also initiated conversations with owners of previous test environments delivered by previous projects to gain evidence as to whether these legacy test environments are available for us to perform testing on them, however feedback thus far is not positive. We do not feel that it would be fair of the reviewers to penalise us for the failure of previous projects to ensure the long term sustainability of their results.

Preservation Scenarios

First new paragraph of page 8 includes the statement: "the quality of information provided is heterogeneous, and not all scenarios provide a link to sample data for testing the scenario. What the deliverable should provide are some proved (i.e. tested) replicable test cases, also extensible to other preservation issues."

We take on board the basis of this recommendation, to provide uniformity to the data quality of the test data collected from preservation scenarios in the expectation that this will provide a set of replicable tests that can be applied to any test environment in a reproducible way so as to provide a fair an independent assessment of that test environment's capabilities.

We would like to make the point that just because a Preservation Scenario's data quality is not homogeneous with all the other Preservation Scenarios acquired to date should not preclude us from capturing details about it. During our work we have come across a number of examples where the actual data cannot be made available for either commercial or legal reasons, but the learning experiences captured in the rest of the Preservation Scenario are still useful, and contain insight as to the approach that was taken by the data owners to preserve that digital material.

The understanding and insight of these processes is potentially more useful than the direct test data itself, as it explains the methods that have been considered, and it is these methods that may be transferable to other domains and data types that can be used by other people to help guide and influence them in their attempts to safeguard their own digital objects.

We have revised Deliverable D14.1 to re-enforce the position that it is the insight gained from learning about the experiences of other Preservation Scenarios that is just as important to the aspiring digital curator as being able to make sets of specific test data available for future review and investigation.

Classification Scheme for Digital Objects

Paragraph 3, page 8 of the review report states: "the proposed classification scheme (pp. 26-27) is based on a few binary characteristics (e.g. static vs. dynamic; passive vs. active) which have been selected without justification. Furthermore, objects in one class do not necessarily have the same characteristics for preservation purposes."

There is no simple solution to how best to classify digital objects for preservation purposes. This issue has been discussed at some length during the work on this WP, and as their was no clear 'winner' in terms of classification scheme to be used we adopted the approach shown in Figure 4 where we include 2 such classification schemes within our assessment matrix. The way the assessment matrix has been constructed allows multiple classification systems to be superimposed on the data so that we can test the suitability of each proposed classification scheme. If the reviewers would like to suggest their own proposal for how to classify digital objects, we would be happy to incorporate this in to our analysis as well.

We have updated the deliverable document to clarify that there is no clear classification scheme 'winner' at this time, and that our work is open to take a wide number of alternative schemes so that any new alternative can be tested using the data set captured to date with minimal impact to any previous work done.

Confusion associated with the matrix assessment

Paragraph 4, page 8 of the review report states: " .... " a more complex version of this matrix assessment approach has been prototyped to demonstrate the current tools and techniques that are in use for preserving digital objects now within the APARSEN community. No access to or documentation of this prototype has been provided. "

We apologies for the confusion that has arisen from this section of text. The author was referring to the initial matrix concept shown in Table 2 of page 22 of D14.1 that had been further developed in to a more complex version as shown in Figure 4, page 26 of the deliverable document. The wording has been updated to better explain the advancement that was originally being alluded too, and toned down a little.

Definition of Testbed and Test Environment

Paragraph 5, page 8, states "There appears to be some ambiguity in use of the term "test environment." In some cases, it appears to mean a specific preservation software environment (e.g. Preservica), while in other cases, it appears to mean a testbed used to test those tools. Clarification on this point would be helpful."

Page 9 of D14.1 included a section relating to the definition of preservation terms relating to testbeds and test environments. We have update this section to clearly differentiate between these two terms. NB: A Testbed is a specialisation of a Test Environment.

Conflict of Interest?

We would like to refute the allegation that Tessella's commercial status in some way limits the work package from achieving its objectives. Tessella are very keen to identify areas of weakness in their archival software solutions so as to allow them to align their future development roadmap to incorporate functionality and features that are of direct need by its archival user base, customer or otherwise. EC project consortia are made up of organisations that provide complementary skills, resources, services and capabilities that when brought together in cooperation allow an objective to be reached that its greater than the sum of its separate parts. Tessella was brought in to this partnership specifically because it has enabling technologies that it can share with this partnership to facilitate a functioning test environment, so to penalise the project for this action would seem absurd.

As per our response to the first point that we raise, we are happy for other emerging test environments to be assessed in a similar way as those that have already been assessed as part of work on WP14, whether these are from research organisations or commercial competitors to Tessella, we seek to create an independent VCoE that makes judgements and recommendations based on evidence, and not on specific project based allegiances. Only that way will we earn and maintain the Trust of the users of the VCoE.

-- AshHunter - 2013-05-23

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2013-06-30 - AshHunter
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback