[Sis-dtn] Re: [Cesg-all] CFDPv1 status update

Tue Jul 14 13:50:44 UTC 2015

My apologies that Scott / Rick beat me to the actual answer…

There’s a lot of discussion in the CESG about ‘reference implementations’; in particular who might produce / host / verify such things, whether an agency would WANT to make their flight code widely available (here’s where my vulnerabilities are!), how to indemnify the producers / distributers, etc.  I concur that automated regression test suites for the interoperability tests would be great to have (and useful for updates, as you mention), but the current requirement is only for a test report that essentially says ‘yes, we tested two independent implementations and they interoperated’.

For LTP I did the interoperability test as a semi-automated test suite, and could presumably reconstitute it if we updated the LTP spec.  Much of the BP testing was done by hand, unfortunately.

If CCSDS were responsible for actually producing software I think it’d be a very different story, but the standards folks are somewhat disconnected from the implementation folks (at least on the NASA side, I suspect in the other agencies as well).  That gap is actually closing a bit at least on the NASA side with some more participation from the flight software folks at Goddard, which seems to be a good thing.

—keith

From: "Kevin. Gifford"
Date: Tuesday, July 14, 2015 at 9:32 AM
To: "Scott, Keith L.", Scott Burleigh, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>", CCSDS All
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>", "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>", "Kevin. Gifford"
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Keith –

Thanks for the kind reply.  I'm now completely onboard with your, Scott's and Rick's points and I appreciate the discussion.  Upon reflection, and looking at the issue from other angles (development vs. missions, agency funding of activities, mission acceptance testing) I am in total agreement as to the sageness of this approach.

I, personally, get hung up on the software engineering observation that if I was writing a standard and responsible for testing interoperability, I'd think that the WG via the two participating primary agencies would produce a reference implementation(s) under standard software development methodologies (revision control, regression testing, etc.).  Then updating, minor or major, would be a case of adding test coverage of the new attributes, rebuilding, and retesting in an effort to provide the soundest specification to stakeholders.  But, as you indicated previously that is a "perfect world" and perhaps this preferential standards development strategy may not always be practical.  So my engineering preference in the CFDPv1 case is an "unrealistic plan is doomed to fail".

This has been an enlightening discussion for me as the Wireless WG is currently producing a Blue Book regarding RFID encoding.  While I expect to follow the reference implementation path for interoperability testing, it is highly probable that there will be downstream modifications required at some point and having the option to reduce the testing burden is a good thing.

Thanks.

Kevin

From: <Scott>, "Keith L." <kscott at mitre.org<mailto:kscott at mitre.org>>
Date: Tuesday, July 14, 2015 7:15 AM
To: Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Burleigh, Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: Re: [Cesg-all] CFDPv1 status update

Kevin,

According to my understanding and my reading of the Organization and Processes manual, the purpose of the interoperability testing is to prove that the specification is unambiguous, not to produce flight-usable code.  Indeed, one of the LTP implementations we used in the interoperability test was in Python — no likely to fly soon :)  The fact that in the case of CFDP NASA is using its ‘flight’ implementation of CFDP is probably a Good Thing, but not required for CCSDS.

I agree that there is concern that the modified bits of the new CFDP spec could have implications that introduce ambiguities in other areas of the specification, but I think that in this case the probability of that is low enough that it doesn’t justify the expense.  The CESG could overrule that.

—keith

I think the operative bits of the of the Organization and Processes manual are:
6.2.6.1.b.1 — Interoperability Testing
A8 — Interoperability is Essential

From: "Kevin. Gifford"
Date: Monday, July 13, 2015 at 5:08 PM
To: "Scott, Keith L.", Scott Burleigh, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>", CCSDS All
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>", "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>"
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Keith –

So the result is the users find and identify any issues arising from independent interoperability testing that are the result of the updated specification?

Is the point of CCSDS interoperability testing to prove the specification is sound/complete or to ensure implementation interoperability?

I'll ask the same question: In the round of updates being discussed in this thread – has any CFDP codebase (NASA or other space agency, e.g., ESA) been or will be modified?

Kevin

From: <Scott>, "Keith L." <kscott at mitre.org<mailto:kscott at mitre.org>>
Date: Monday, July 13, 2015 2:56 PM
To: "Burleigh, Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>, Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: Re: [Cesg-all] CFDPv1 status update

I agree with Scott.  In a perfect world, we’d have an automated regression test suite left over from the original testing (or follow-on work), and just re-run it.  In practice it is unfortunately not that simple, and since the point of the tests is really to prove out that the spec is sound rather than to test the implementation(s), I think we’ll need to stick with the smaller test if we can do it.

—keith

From: Scott Burleigh
Date: Monday, July 13, 2015 at 4:12 PM
To: "Kevin. Gifford", "Scott, Keith L.", "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>", CCSDS All
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>", "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>"
Subject: RE: [Cesg-all] CFDPv1 status update

Kevin, FWIW, I disagree.  I don’t think you have stated here what the “clear need” for repeating the interoperability testing of all parts of CFDP might be.

The purpose of this interoperability testing is not to validate the correct operation of protocol implementations.  It is rather to validate the clarity and completeness of the protocol specification – demonstrating that the document provided sufficient guidance to enable independent implementers to develop software that interoperates.

The clarity and completeness of the original, unmodified portions of the specification was validated many years ago, in the extensive interoperability testing that preceded publication of the CFDP Blue Book, and it remains valid because those portions of the specification have not been revised.

What has not yet been validated is the revised text of the specification, and that is what requires interoperability testing now.  I don’t think additional testing beyond what is required to prove out the specification can be justified.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 9:55 AM
To: Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>; CCSDS All
Cc: Burleigh, Scott C (312B); sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>; sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Keith –

From a pedantic test engineering standpoint, I'd strongly suggest regression testing the entire CFDP protocol suite.

Certainly, I am not in-the-loop for the detailed technical protocol updates, but unless the updates were very minor, I see no engineering reason for not testing/retesting/regression-testing the entire updated (and perhaps substantially roto-tilled) codebase.  If the decision is a funds-available business decision, and the updates are very minor, then perhaps with explicit justification the additional testing could be either reasonably (defendable by some reviewed document) waived or possibly postponed.

I understand this increases the testing burden and necessitates (most likely) the need for additional resources.  But if CFDP is used on any current (or future) spacecraft, and especially the ISS in the near-term, then I think not testing the entire updated protocol suite is an engineering mistake (not performing due diligence when there is a clear need).

I just don't see now not retesting the entire codebase can be justified unless the updates are very, very minor.  Am I missing something?

Kevin

From: <Scott>, "Keith L." <kscott at mitre.org<mailto:kscott at mitre.org>>
Date: Monday, July 13, 2015 10:34 AM
To: "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Subject: [Cesg-all] CFDPv1 status update

All,

The CFDPv1 book completed agency review<http://public.ccsds.org/sites/cwe/rids/Lists/CCSDS%207270P41/Overview.aspx>, and the changes necessitated by the RIDs were minor enough to not warrant another agency review and have been resolved.  The attached book with resolutions, and RID resolution spreadsheet, are for your information.

The working group is going to interoperability test the NEW parts of the protocol only — and that work is under way.  When the interoperability testing is complete SIS will issue a resolution to submit the book for final CESG Poll and then CMC Poll; that resolution will be accompanied by the final version of the book (hopefully same as attached) and the interoperability test report for the new elements of the protocol.

As just mentioned, to reduce the interoperability testing burden, we plan to interoperability test only the NEW portions of CFDP (capabilities added by the current update).  When we submit the final document and interoperability test report, SIS will essentially be requesting a waiver on testing the remaining parts of the protocol.  If anyone knows now that they’ll have an issue with this approach, could you please let me know?

V/r,

—keith

From: Scott Burleigh
Date: Monday, July 13, 2015 at 12:16 PM
To: "Scott, Keith L."
Subject: RE: We're ready to go with a resolution to publish the CFDPv1 Blue Book, right?

I think we can assert that none of the technical changes introduced since the agency review affect protocol functionality profoundly enough to warrant the expense of an additional agency review; all of the changes are minor details.  That’s going to be a judgment call for CESG, I guess, but I think it’s a defensible assertion.

Scott

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/sis-dtn/attachments/20150714/dcf311ec/attachment.html>