[Moims-dai] Action from MOIMS-DAI Webex 20180605

Mon Jun 11 21:55:01 UTC 2018

See below

From: MOIMS-DAI <moims-dai-bounces at mailman.ccsds.org> On Behalf Of Mark Conrad
Sent: 11 June 2018 22:04
To: MOIMS-Data Archive Interoperability <moims-dai at mailman.ccsds.org>
Subject: Re: [Moims-dai] Action from MOIMS-DAI Webex 20180605

See comments below.

Mark Conrad
NARA Information Services

Systems Engineering Division (IT)
The National Archives and Records Administration
Erma Ora Byrd Conference and Learning Center
Building 494, Room 225
610 State Route 956
Rocket Center, WV  26726

Phone: 304-726-7820
Fax: 304-726-7802
Email: mark.conrad at nara.gov <mailto:mark.conrad at nara.gov>  

On Mon, Jun 11, 2018 at 5:26 AM, david <david at giaretta.org <mailto:david at giaretta.org> > wrote:

Action from Webex 20180605

1.       David to write an example of the application of OAIS preservation concepts to components of PDI including illustration of Ingest and recursion.

A Producer – in this case outside the OAIS A Producer is a role that can be played by an internal or external source. However the role is always outside the OAIS. See Figure 2-1 and Section 2-1 "Outside the OAIS are Producers, Consumers, and Management." – wishes to submit information to be preserved by the OAIS. Agreement has been reached on the procedures to be followed. The Producer prepares one or more SIPs which are transferred to the OAIS.

Perhaps I should have made it clearer – in this case I am referring the entity which is playing the role of Producer is external to the OAIS. See the Glossary definition off Producer.

* A Receive Submission function receives the SIP, and acknowledges the receipt or requests a re-submission if there is an error, for example as reported by Quality Assurance.

* The SIP (or the collection of SIPs together) contains the Content Information (Data Object and Representation Information), plus PDI and may also provide Information Property Descriptions of Information Properties. According to the current definition, Information Properties (Description?) is part of the Content Information.

* The AIP is generated by a Generate AIP function from the SIP(s) This is not specific enough to evaluate your proposal. What is in the SIPs? How are they transformed/mapped to produce the AIP? with any additional Representation Information required Isn't the additional Representation Information part of the AIP?. I think that logically we can say that it may also produce Fixity Information If you are forming an AIP this is required., as part of verifying the transfer or if some file format conversions have taken place. Additional Provenance should be created to record the fact that the information has been taken into the OAIS Required if you are creating an AIP.  Some additional Context may be created Required if you are making an AIP/ and Reference Information will also be created to allow the Content Information to be found. The Packaging Information will be determined by the structure of the AIP which the OAIS has decided to use. Note that an AIP is defined as a logical container.

* The associated Descriptive Information is created by Generate Descriptive Information.

* Coordinate Updates transfers the AIP to Archival Storage and the Descriptive Information to Data Management. 

All the above was taken more or less from OAIS itself, apologies if I have garbled it a little – check the standard.

The OAIS will need to preserve the PDI. Let us see how the each of the components of PDI is preserved. Let us say that the Fixity is encoded as a hash code Are you assuming all of the PDI is in a single file?, for example the Data for which is a sequence of bits which encode ASCII characters. 

No I am not assuming the PDI is a single file. The Fixity I am referring to in the above is the Fixity of the Content above, not the Fixity of the PDI, which I discuss below. 

The following runs through a likely process, mapping it to the Ingest Functional Entity.

A Producer – in this case inside the OAIS No. A Producer is a role that can be played by an internal or external source. However the role is always outside the OAIS. See Figure 2-1 and Section 2-1 "Outside the OAIS are Producers, Consumers, and Management." – wishes to submit information to be preserved by the OAIS. Agreement has been reached on the internal procedures to be followed. The Producer prepares one or more SIPs which are transferred to the OAIS - in this case this is an internal transfer. No. Producers are outside the OAIS so their submissions come from outside the OAIS.

Check the Glossary: Producer:  The role played by those persons or client systems that provide the information to be preserved.  This can include other OAISes or internal OAIS persons or systems.

* In this case the SIP would contain the hash code. 

*       The Representation Information will probably be the same for many examples of Fixity if the OAIS uses the same method for many pieces of data; for example the RepInfo could be (1) Structure: ASCII code to go from bits (2) Semantic: SHA256 was used to create the hash. Therefore there needs only to be a pointer to the RepInfo for the hash. 

*       The Provenance would be that this was created at a particular time, who created it as part of what business proccess and what file the hash is associated with. . The hash would have been created either by the external Producer or the OAIS. In the latter case one would only need to point to a single statement that the OAIS created the hash. No. You also need to know when it was created and what business process included the generation of the hash value, what file it was generated from, etc.

You need to read the sentences together. I had hoped to make it clear that there are likely to common features, such as the statement about who created the Provenance, and as you say, what the business process used was etc, and there will be something which is always specific, namely the time.

*       The Reference Information provides the location where the hash may be retrieved – unique for each hash – surely each hash will have such a reference otherwise how can it be retrieved?

*       The Access Rights would again be the same for things that are being preserved – no-one is allowed to alter the hash. 

*       The Context would be that the hash is created as part of the OAIS preservation functionality.

All the things (RepInfo, Provenance creator, Access Rights, Context) which are the same for all, or lots of similar, things, could be contained in the documentation about the process. What does this mean?

It means that the process is documented, as the OAIS’s internal processes should be.

* The Receive Submission (possibly different from the one used for SIPs from the external Producer) function receives the SIP, and acknowledges the receipt or requests a re-submission if there is an error, for example as reported by Quality Assurance. One might expect these may be taken care of by the normal computing infrastructure since it is an internal transfer. No. All submissions are treated as external submissions. See Section 2.1.

See the Glossary as noted above

* The SIP(s) contains the Content Information (Data Object and Representation Information), plus PDI and may also provide Information Property Descriptions of Information Properties. 

* The AIP is generated by a Generate AIP function from the SIP(s) with any additional Representation Information required.You are not providing sufficient detail to evaluate your proposal.  Again the Generate AIP function may be different from the one used for information from n external Producer. It should also produce Fixity Information, as part of verifying the transfer or if some file format conversions have taken place. Additional Provenance should be created to record the fact that the information has been taken into the OAIS. Some additional Context may be created and Reference Information will also be created to allow the Content Information to be found. All the elements you have identified in the previous sentences are mandatory if you are generating an AIP. The Packaging Information will be determined by the structure of the AIP which the OAIS has decided to use. Note that an AIP is defined as a logical container.

o As noted above, many of the components of the AIP would be common to many things and could be covered by a procedures document What is this procedures document in OAIS terminology? . The Packaging Information could be contained in a procedures document which essentially just has to say where all the components of the AIP are to be found.

* The associated Descriptive Information is created by Generate Descriptive Information.

o This again could be covered by a procedures document.

* Coordinate Updates transfers the AIP to Archival Storage and the Descriptive Information to Data Management. 

I think I have provided enough detail – to provide more would be to stray into implementation details. 

OAIS does not have a specific term for the documentation I am referring to. Just as it does not have a specific name for most documents.

As the next level of iteration one can work through the RepInfo or PDI of the procedures documents mentioned above – for example a Word file, written in English, written by a member of the OAIS staff and finalised at a specified time.

I hope it can be understood that the view that I take, re-using OAIS Information concepts repeatedly, does not lead to a plethora of confusing AIPs See my questions below. but instead allows us to be logically coherent in our approach to preservation by simply mapping what I would imagine any good repository would normally be doing, or, alternatively, to check that a repository _is_ doing the right things. To confirm this all you have to do is to go through the above steps and ask yourself whether any steps (or some equivalent steps since we are implementation agnostic) can be missed out.

How do you map the AIPs created by the OAIS in the role of the Producer to the components of the first AIP generated from the original Content Information? For example, if the the Fixity Information in the second AIP generated is the Fixity Information for one or more components of the original AIP, how is that reflected in the original AIP? How is this mapping created for each of the subsequent iterations of AIP generation? 

The Packaging Information provides the linkages.

Regards

David

_______________________________________________
MOIMS-DAI mailing list
MOIMS-DAI at mailman.ccsds.org <mailto:MOIMS-DAI at mailman.ccsds.org> 
https://mailman.ccsds.org/cgi-bin/mailman/listinfo/moims-dai

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/moims-dai/attachments/20180611/c3ce3348/attachment.html>