[Sis-dtn] Re: [Cesg-all] CFDPv1 status update

[Scott, Keith L.]

Kevin,

According to my understanding and my reading of the Organization and Processes manual, the purpose of the interoperability testing is to prove that the specification is unambiguous, not to produce flight-usable code.  Indeed, one of the LTP implementations we used in the interoperability test was in Python — no likely to fly soon :)  The fact that in the case of CFDP NASA is using its ‘flight’ implementation of CFDP is probably a Good Thing, but not required for CCSDS.

I agree that there is concern that the modified bits of the new CFDP spec could have implications that introduce ambiguities in other areas of the specification, but I think that in this case the probability of that is low enough that it doesn’t justify the expense.  The CESG could overrule that.

—keith


I think the operative bits of the of the Organization and Processes manual are:
6.2.6.1.b.1 — Interoperability Testing
A8 — Interoperability is Essential



From: "Kevin. Gifford"
Date: Monday, July 13, 2015 at 5:08 PM
To: "Scott, Keith L.", Scott Burleigh, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>", CCSDS All
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>", "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>"
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Keith –

So the result is the users find and identify any issues arising from independent interoperability testing that are the result of the updated specification?

Is the point of CCSDS interoperability testing to prove the specification is sound/complete or to ensure implementation interoperability?

I'll ask the same question: In the round of updates being discussed in this thread – has any CFDP codebase (NASA or other space agency, e.g., ESA) been or will be modified?

Kevin

From: <Scott>, "Keith L." <kscott at mitre.org<mailto:kscott at mitre.org>>
Date: Monday, July 13, 2015 2:56 PM
To: "Burleigh, Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>, Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: Re: [Cesg-all] CFDPv1 status update

I agree with Scott.  In a perfect world, we’d have an automated regression test suite left over from the original testing (or follow-on work), and just re-run it.  In practice it is unfortunately not that simple, and since the point of the tests is really to prove out that the spec is sound rather than to test the implementation(s), I think we’ll need to stick with the smaller test if we can do it.

—keith

From: Scott Burleigh
Date: Monday, July 13, 2015 at 4:12 PM
To: "Kevin. Gifford", "Scott, Keith L.", "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>", CCSDS All
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>", "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>"
Subject: RE: [Cesg-all] CFDPv1 status update

Kevin, FWIW, I disagree.  I don’t think you have stated here what the “clear need” for repeating the interoperability testing of all parts of CFDP might be.

The purpose of this interoperability testing is not to validate the correct operation of protocol implementations.  It is rather to validate the clarity and completeness of the protocol specification – demonstrating that the document provided sufficient guidance to enable independent implementers to develop software that interoperates.

The clarity and completeness of the original, unmodified portions of the specification was validated many years ago, in the extensive interoperability testing that preceded publication of the CFDP Blue Book, and it remains valid because those portions of the specification have not been revised.

What has not yet been validated is the revised text of the specification, and that is what requires interoperability testing now.  I don’t think additional testing beyond what is required to prove out the specification can be justified.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 9:55 AM
To: Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>; CCSDS All
Cc: Burleigh, Scott C (312B); sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>; sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Keith –

From a pedantic test engineering standpoint, I'd strongly suggest regression testing the entire CFDP protocol suite.

Certainly, I am not in-the-loop for the detailed technical protocol updates, but unless the updates were very minor, I see no engineering reason for not testing/retesting/regression-testing the entire updated (and perhaps substantially roto-tilled) codebase.  If the decision is a funds-available business decision, and the updates are very minor, then perhaps with explicit justification the additional testing could be either reasonably (defendable by some reviewed document) waived or possibly postponed.

I understand this increases the testing burden and necessitates (most likely) the need for additional resources.  But if CFDP is used on any current (or future) spacecraft, and especially the ISS in the near-term, then I think not testing the entire updated protocol suite is an engineering mistake (not performing due diligence when there is a clear need).

I just don't see now not retesting the entire codebase can be justified unless the updates are very, very minor.  Am I missing something?

Kevin

From: <Scott>, "Keith L." <kscott at mitre.org<mailto:kscott at mitre.org>>
Date: Monday, July 13, 2015 10:34 AM
To: "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Subject: [Cesg-all] CFDPv1 status update

All,

The CFDPv1 book completed agency review<http://public.ccsds.org/sites/cwe/rids/Lists/CCSDS%207270P41/Overview.aspx>, and the changes necessitated by the RIDs were minor enough to not warrant another agency review and have been resolved.  The attached book with resolutions, and RID resolution spreadsheet, are for your information.

The working group is going to interoperability test the NEW parts of the protocol only — and that work is under way.  When the interoperability testing is complete SIS will issue a resolution to submit the book for final CESG Poll and then CMC Poll; that resolution will be accompanied by the final version of the book (hopefully same as attached) and the interoperability test report for the new elements of the protocol.

As just mentioned, to reduce the interoperability testing burden, we plan to interoperability test only the NEW portions of CFDP (capabilities added by the current update).  When we submit the final document and interoperability test report, SIS will essentially be requesting a waiver on testing the remaining parts of the protocol.  If anyone knows now that they’ll have an issue with this approach, could you please let me know?

V/r,

—keith


From: Scott Burleigh
Date: Monday, July 13, 2015 at 12:16 PM
To: "Scott, Keith L."
Subject: RE: We're ready to go with a resolution to publish the CFDPv1 Blue Book, right?

I think we can assert that none of the technical changes introduced since the agency review affect protocol functionality profoundly enough to warrant the expense of an additional agency review; all of the changes are minor details.  That’s going to be a judgment call for CESG, I guess, but I think it’s a defensible assertion.

Scott

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/sis-dtn/attachments/20150714/1b1115cf/attachment.html>