[Moims-dai] [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Shames, Peter M (US 312B)
peter.m.shames at jpl.nasa.gov
Fri Jul 10 23:14:22 UTC 2020
Dear David,
Thank you for your patience with me, it has taken a while for me to get to the point where I could devote an unbroken stretch of several hours to this document, the PIDs I submitted, and your proposed dispositions of them. I have now gone through in detail, the revised document, the set of PIDs, and the dispositions that the DAI WG has proposed.
When I read the document initially I proposed a set of what I thought were simple extensions that would turn the document into a really useful reference architecture, which is what I understood to be what you intended. A reference architecture, in my view, may be highly abstract, i.e. not at all suitable for direct implementation, but at the same time sufficiently clear that the reader can understand the complex technical concepts and how they are intended to fit together.
CCSDS has its own definitions for normative track Magenta Books, which include reference architecture, which can be found in sec 6.1.4.3 of CCSDS Org & Proc, A02.1-Y-4. This document is titled a "Reference Model for an Open Archival Information System". As such, and given that it uses the word architecture in a number of places, I would expect it to comply with the norms for "reference architecture". A useful definition of that is found in Wikipedia, https://en.wikipedia.org/wiki/Reference_architecture. That source also has a definition for "reference model", https://en.wikipedia.org/wiki/Reference_model, but after reviewing that, and what is intended to be covered in the OAIS, I think the term reference architecture is closer.
I am sorry to have to say this, but by neither test do I think that the OAIS document meets the mark.
After looking again through the set of PIDs and dispositions I found the one that was most telling, to my mind. In PID SEA-650x0-024 I raised the issue of "conformance" and stated that I thought it was weakly handled. The response is that the only conformance element was sec 2.3. Here is the statement that was made:
Reject the recommendation of adding conformance with the key section (sec 4) of the
document and also the other, related, edits identified elsewhere in this set of RIDs.
The Functional Model has always been excluded from the conformance statement because
OAIS is not a design document.
We think the use of the conformance section [Sec 2.3] has been very useful in the reference
model as it stands.
The "To:" text in this RID is obviously not seriously being proposed.
It seems that all of the recommended changes were treated in a similarly cavalier fashion. i.e treated as "obviously not seriously being proposed". Furthermore, in sec 2.3 the only thing that could in any way be considered as "suitable for conformance validation" is the provision of some terminology (which is not in Sec 2.3) and the very abstract info model in fig 2-3 and some equally abstract data flows in fig 2-4. By this "conformance" test almost any system would "comply", and at that point the assessment of this as a normative Magenta Book falls apart.
Since it is clearly the wish of the DAI WG to leave this document as it is, and not to improve the contents so as to elevate it to the level of a normative Magenta Book, I am afraid that my recommendation is to request that it be edited into Green Book format and published in that form.
I want you to know that I did not arrive at this point lightly. I spent a lot of time reviewing the PIDs, the responses, and the document. I'd like it to be otherwise. But after due consideration of all of the inputs and feedback this is the only conclusion that I think is acceptable.
I am, of course, open to further discussion.
Very best regards, Peter
From: David Giaretta <david at giaretta.org>
Date: Tuesday, March 24, 2020 at 5:30 AM
To: Peter Shames <peter.m.shames at jpl.nasa.gov>
Cc: CCSDS Secretariat <secretariat at mailman.ccsds.org>, MOIMS-DAI List <moims-dai at mailman.ccsds.org>
Subject: RE: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Dear Peter
Thank you very much for the collection of RIDS.
We have spent the last two months reviewing and discussing those RIDS and have made changes to OAIS as a result. I provide below the link to the CWE file which has mark-up with references to the applicable RIDs. I also attach the link to the completed RID file with our responses.
Updated OAIS: https://cwe.ccsds.org/moims/docs/MOIMS-DAI/Draft%20Documents/OAIS%20v3/650x0020_CESG_Aproval_Mod200211-SEA.doc?Web=1
RIDS: https://cwe.ccsds.org/moims/docs/MOIMS-DAI/Draft%20Documents/OAIS%20v3/CESG-P-2019-12-007_PID_650x0-SEA-v1%20(1)-responses.txt
As you will see, we had concerns about some RIDS things as changing the diagram conventions but other comments made us clarify our text and include new ideas, thereby improving the document, for which we are very grateful.
The actual extensive discussions took place in our weekly teleconferences and the results are on the review site – see http://review.oais.info/buglist.cgi?bug_status=__all__&content=SEA-650x0&list_id=2647&order=bug_id&product=OAIS%20June%202012&query_format=specific.
We believe these resolve the RIDS.
Regards
..David
From: Shames, Peter M (US 312B) <peter.m.shames at jpl.nasa.gov>
Sent: 14 January 2020 23:38
To: david at giaretta.org
Cc: 'John Garrett' <garrett at his.com>; Hughes, John S (US 398B) <john.s.hughes at jpl.nasa.gov>
Subject: Re: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi David,
As I said "as so often occurs in email "discussions" we seem to be not quite understanding one another".
Regardless, I hope that the RIDs I submitted are clear enough in their intent. And I will make myself available to discuss them if you think that is warranted.
I think these will arrive "officially" in a day or two, but here they are now.
Very best regards, Peter
From: David Giaretta <david at giaretta.org<mailto:david at giaretta.org>>
Date: Tuesday, January 14, 2020 at 2:19 PM
To: Peter Shames <peter.m.shames at jpl.nasa.gov<mailto:peter.m.shames at jpl.nasa.gov>>
Cc: John Garrett <garrett at his.com<mailto:garrett at his.com>>, Steve Hughes <john.s.hughes at jpl.nasa.gov<mailto:john.s.hughes at jpl.nasa.gov>>
Subject: RE: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi Peter
We _do_ look forward to receiving your comments. I certainly did not mean that I want to reject your suggestions without looking in detail at them.
My email was not trying to say that we want to “stick to our guns”. I was trying to say that I don’t think we ignored or did not consider the sort of things you were talking about. We just had a different plan, which we are working on. If that plan was not working then we should change the plan.
Moreover we did recognise the “fast paced technology” changes from the start, and the reason that OAIS has survived and been so widely used over the 2 decades is that we got the level of abstraction about right.
Cheers
..David
From: Shames, Peter M (US 312B) <peter.m.shames at jpl.nasa.gov<mailto:peter.m.shames at jpl.nasa.gov>>
Sent: 14 January 2020 21:13
To: david at giaretta.org<mailto:david at giaretta.org>
Cc: 'John Garrett' <garrett at his.com<mailto:garrett at his.com>>; Hughes, John S (US 398B) <john.s.hughes at jpl.nasa.gov<mailto:john.s.hughes at jpl.nasa.gov>>
Subject: Re: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi David,
Well, as so often occurs in email "discussions" we seem to be not quite understanding one another. I know that you guys aim for the "gazillion different archives with a gazillion squared types of digital objects". I do not think that I used the words "space" or "agencies" once, so let's set that aside. Everything I am suggesting is neutral to space agencies, archive communities, libraries, etc etc, and to specifics of implementation architectures. It is intended to all be abstract architecture, and also technology neutral, but viable when rendered in a variety of different real implementation / deployment architectures.
From my PoV, the validity of my remarks on the "conformance" aspects of these very abstract concepts (in sec 2.3 and 3.2) remain. It's actually rather remarkable that anyone could think that you could assess conformance with these very abstract concepts, but if it works for some community I am not going to stand in the way, or even attempt to.
As for the length of the Functional Model section, I was thinking that this included the whole of sec 4, which does seem to be 56 pages in length. I'll grant you that the Information Model parts could be thought of as separate from the Functional Model, but it is all abstract architecture as far as I am concerned. And before you make it smaller you might just consult your community, I think it may be one of the most useful parts in terms of something that anyone could "conform" to. But that's just my opinion and I may be totally wrong.
I'll still provide my feedback. Feel free to review it, accept it, reject it, or shove it into some other TBD document.
Cheers, Peter
PS – Using "we started this way 20 years ago and we're sticking to our guns" as an argument for not evolving, in this age of fast paced technology, seems like a sure-fire recipe for obsolescence.
From: David Giaretta <david at giaretta.org<mailto:david at giaretta.org>>
Date: Tuesday, January 14, 2020 at 12:28 PM
To: Peter Shames <peter.m.shames at jpl.nasa.gov<mailto:peter.m.shames at jpl.nasa.gov>>
Cc: John Garrett <garrett at his.com<mailto:garrett at his.com>>, Steve Hughes <john.s.hughes at jpl.nasa.gov<mailto:john.s.hughes at jpl.nasa.gov>>
Subject: RE: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi Peter
What you describe is a perfectly possible way to go with the book.
However the approach we actually took, and planned out from the start, about 2 decades ago, was rather different, and has, as far as I can see, has been pretty successful in terms of take-up.
One way to think about this is that rather than having a target audience of a few dozen space agencies and a few thousand projects, we had a gazillion different archives with a gazillion squared types of digital objects. Moreover these archives were saying that they could preserve things but we were sure that, in particular, they could not preserve space data, or indeed any other data. Our aim was to fix that by getting a wide take up of the standard. Otherwise there was a danger that, as Vint Cerf says, we could enter a digital dark age.
That is why the Information Model, and the Mandatory Responsibilities, are the focus of conformance. It seems to work and forms the basis of ISO 16363 and the audit and certification process.
We probably did write too much about the Functional Model, but there are really only about 20 pages on it. Not sure where you get 50+.
We recognised that we could not cover everything in OAIS so we included a roadmap of things to be developed later, which now we have moved to an external document to allow a more rapid evolution. In 2002 we had:
· standard(s) for the interfaces between OAIS type archives;
· standard(s) for the submission (ingest) methodology used by an archive;
· standard(s) for the submission (ingest) of digital data sources to the archive;
· standard(s) for the delivery of digital sources from the archive;
· standard(s) for the submission of digital metadata, about digital or physical data sources, to the archive;
· standard(s) for the identification of digital sources within the archive;
· protocol standard(s) to search and retrieve metadata information about digital and physical data sources;
· standard(s) for media access allowing replacement of media management systems without having to rewrite the media;
· standard(s) for specific physical media;
· standard(s) for the migration of information across media and formats;
· standard(s) for recommended archival practices;
· standard(s) for accreditation of archives.
Gradually these standards have been created by us or by others. If we had tried to shoehorn more into OAIS we would probably never have finished and if we had finished then it would not have been taken up very widely.
You can see that Steve’s work fits into this grand scheme from so long ago. And that plan seems to be working so we would like to stick to it.
I hope that explains why we took the path that we did take and how we had planned from the start to put all these other points, which are certainly all essential for implementations, elsewhere.
Cheers
..David
From: Shames, Peter M (US 312B) <peter.m.shames at jpl.nasa.gov<mailto:peter.m.shames at jpl.nasa.gov>>
Sent: 14 January 2020 19:18
To: david at giaretta.org<mailto:david at giaretta.org>
Cc: 'John Garrett' <garrett at his.com<mailto:garrett at his.com>>
Subject: Re: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi David,
Thanks for the quick reply. I was actually sitting here looking at my inbox and wondering if/when I was going to hear from you when this reply popped up. Fortuitous. I still have a couple of topics that I think ought to be addressed in the RIDs, including security, so I'll get those out to you later today.
Your assumption is quite correct in that a lot of my focus has been on the Functional Model, since that seems to really be the heart of the document, at 50+ pages in length. I do recognize that a lot of what you are doing is providing terminology, rationale, and motivation, but it seems to me that the heart of the document, in terms of the amount of effort expended, is actually in this Functional Model reference architecture. And that presenting those concepts in as clear a fashion as possible is really one of the most valuable things you can offer. I wonder if your community of users would agree with this?
I was struck by your use of the term "conformance", so I went to see what the document actually says. Conformance to a reference model is a bit of a tricky thing, since it is all abstractions and there is not much "there, there" to tie conformance to. It seems that the notion of conformance is currently tied to the "information model" in sec 2.3, a rather loose set of abstractions, and to the "responsibilities" in Sec 3.2, also a rather loose set. You could have chosen to provide a stronger, but still abstract, definition of the AIP, SIP, DIP, etc but this was not done. The result is that there are a whole lot of words where some models and explanations might have been clearer. How do you assess conformance to a very abstract information model and to these "responsibilities"? And why isn't a similar "conformance" assessment to an abstract Functional Architecture just as valid an assessment?
I was also struck by the idea that what may be most useful is that OAIS "compliant" archives "can and do use that common terminology to clearly describe their systems so they can compare themselves to other archives". Is this purpose of comparison, in and of itself, really that valuable, or is what is truly valuable in the OAIS the set of concepts for thinking about:
1) What an archive must do to preserve, and serve, data
2) How it does those tasks
3) How it might be constructed to do those tasks
4) How it describes the information that it manages
5) How it provides access to that information
6) How it ensures the integrity of that information, in both an immediate and a long term sense
7) What kinds of access is provided and what products are made available to the users
8) And, how information from more than one archive can be commingled to produce products that one alone cannot support
I am sure that there are other topics that are covered, but these seem really essential to me. Did I get this wrong?
I'm really not trying to pick apart what you guys have done, rather I'm looking at it perhaps through a different set of lenses. I think there is value in this, especially if it is enhanced in certain ways. I think it could provide a much stronger description of the reference Functional Model and Information Model, even if it is still all abstractions, and that doing that would make the document even more valuable. I hope you guys get what I am proposing.
Enough for now. I think this provides useful background for my motivation for the list of RIDs I'll shortly be sending along.
Very best regards, Peter
From: David Giaretta <david at giaretta.org<mailto:david at giaretta.org>>
Date: Tuesday, January 14, 2020 at 10:37 AM
To: Peter Shames <peter.m.shames at jpl.nasa.gov<mailto:peter.m.shames at jpl.nasa.gov>>
Cc: John Garrett <garrett at his.com<mailto:garrett at his.com>>
Subject: [EXTERNAL] RE: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi Peter
Thanks for the heads-up.
Would it be possible for you to send me the RIDS you have now so we can make an assessment of what is involved?
As you say, we do not want to make this a Blue Book. From what you wrote you may be mostly looking at the Functional Model, which is not part of the conformance, so we can be a bit freer with it. The purpose of the Functional Model is to provide a lot of terminology, most of which dates back to the first version in 2002. Archives, whether OAIS conformant or not, can and do use that common terminology to clearly describe their systems so they can compare themselves to other archives. However as we state several times, we do not want the book to be mistaken for a design for an archive.
Thanks for the offer of help, which we may take up once we have seen the RIDS.
Regards
..David
From: Shames, Peter M (US 312B) <peter.m.shames at jpl.nasa.gov<mailto:peter.m.shames at jpl.nasa.gov>>
Sent: 13 January 2020 21:08
To: David Giaretta <david at giaretta.org<mailto:david at giaretta.org>>
Subject: CESG conditions on your OIAS Reference Model, CCSDS 650.0-P-2.0
Hi David,
I just want to alert you that as I started reading this document as part of the CESG review that I, almost immediately as it turned out, stumbled over some issues of terminology and content that I thought needed to be addressed. These were significant enough that they tipped me into "give this a serious reading" mode. Right now I am on pg 71 and I have thirteen (13) RIDs queued up. Some of them are going to take a bit of work to fix, so I wanted to alert you.
One of them, which I have not yet written up, concerns the approach that you have used vis-à-vis the definition of OAIS services and service interfaces. This is a Magenta Book, which gives you some amount of latitude in just what is included. As such, this is mostly about procedures and processes, but you do define a sort of abstract data system model, with functional elements, service interfaces (some internal and some external), and data flows (some with named data objects with their own abstract definitions like AIP, SIP, and DIP). All of this is abstract, and it is fine that this is the case in a Magenta Book. If they were concrete definitions you would be in Blue Book territory and I believe that you do not want to go there.
That said, I do think that you are missing a fine opportunity to provide some further, and very useful in my opinion, abstract definitions of these service interfaces. The work that was published in the CCSDS Reference Architecture for Space Information Management (RASIM), CCSDS 312.0-G-1 (attached for reference), provides a set of straightforward, but still abstract, functional interfaces for exactly the sorts of registry, repository, query, etc interfaces that this OAIS model mentions, but does not clearly define. I happen to be of the opinion that including such definitions for these fundamental functional (but abstract) building block objects, and for service interfaces (also abstract) that reference them would be a huge improvement in the document. Now, it is true that I am biased, that work occurred in a WG in my Area and I contributed to it, but I also think I am being objective when I suggest that your document would be significantly enhanced in value by these technical, but abstract, additions.
I would even go so far as to propose that I help your WG in making these additions to the document. That's up to you and your team, of course, but it is a sincere offer.
That said, I will be bringing all of these concerns to your WG in the form of Conditions on publication. What you choose to do with them, and how you disposition them, is really your business and this is neither a threat nor a bribe, just an honest expression of concern and a set of technical issues to be discussed and resolved.
I am happy to discuss any of this, either now or after I submit the CESG conditions.
Best wishes, Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/moims-dai/attachments/20200710/a081de89/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SEA-650x0-PIDs resultion 10Jul20.pdf
Type: application/pdf
Size: 244592 bytes
Desc: SEA-650x0-PIDs resultion 10Jul20.pdf
URL: <http://mailman.ccsds.org/pipermail/moims-dai/attachments/20200710/a081de89/attachment-0002.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 650x0020_CESG_Approval_Mod200211-SEA-10Jul20.pdf
Type: application/pdf
Size: 4931645 bytes
Desc: 650x0020_CESG_Approval_Mod200211-SEA-10Jul20.pdf
URL: <http://mailman.ccsds.org/pipermail/moims-dai/attachments/20200710/a081de89/attachment-0003.pdf>
More information about the MOIMS-DAI
mailing list