[Sis-dtn] RE: [Cesg-all] CFDPv1 status update

[Burleigh, Scott C (312B)]

Kevin, yes, exactly - the entire text of the Bundle Protocol spec needed verification by interoperability testing, because none of it had previously been verified; from the perspective of CCSDS it was all new, and all new normative specification text requires interoperability testing.  And the same is true for the CFDP v1 specification: all of the new normative specification text requires interoperability testing.  It's really the same level of testing requirement, just a different scope due to the difference in prior validation

And yes, absolutely, this interoperability testing doesn't guarantee the mission usability of any of the implementations involved.  That acceptance testing still needs to happen.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 4:33 PM
To: Burleigh, Scott C (312B); Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org; CCSDS All
Cc: sis-cfdp at mailman.ccsds.org; sis-dtn at mailman.ccsds.org
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Scott -

OK, for DTN as a brand new specification there was a different level of interoperability testing required as compared to CFDPv1 then?

I'm OK with that with the condition that "mission" interoperability testing is expected to transpire downstream.

Thanks for taking the time to discuss.

Kevin

From: <Burleigh>, "Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>
Date: Monday, July 13, 2015 5:25 PM
To: Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Scott, Keith L (9730-Affiliate)" <kscott at mitre.org<mailto:kscott at mitre.org>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: RE: [Cesg-all] CFDPv1 status update

Hi, Kevin.  Actually, we have not yet undertaken any of the interoperability testing for the CFDPv1 (Revisions) Pink/Blue Book.  I sent out an email a few days ago listing the tasks involved and asking for volunteers, and I am sure I will be hearing some responses shortly.  Notionally, though, I expect the testing will be done using modified versions of the ION (NASA) and ESA implementations of CFDP.

Keith is the real authority for your second question, but my own answer would be yes, this testing was required in order to get the Bundle Protocol specification published as a Blue Book.  The specification was a brand-new CCSDS specification, never previously verified by CCSDS interoperability testing, so that verification was needed.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 4:17 PM
To: Burleigh, Scott C (312B); Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>; CCSDS All
Cc: sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>; sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>
Subject: Re: [Cesg-all] CFDPv1 status update

Scott -

Thank you for your reply.

I understand your explanation below.

For my information, which agencies have been involved in the CFDP interoperability testing that has been done to to this point on CFDPv1?

As a possibly related activity, when JAXA and NASA completed ECOS BP interoperability testing at Pasadena in March 2015 was this testing necessary for the Blue Book publication process?

Kevin

From: <Burleigh>, "Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>
Date: Monday, July 13, 2015 3:55 PM
To: Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Scott, Keith L (9730-Affiliate)" <kscott at mitre.org<mailto:kscott at mitre.org>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: RE: [Cesg-all] CFDPv1 status update

Kevin, sorry, I wasn't trying to sidestep your question: yes, certainly implementations have been modified.

This has been done (at this point) solely for the purpose of testing the specification revisions, by demonstrating interoperability between two different implementations of the same specification revision text.

The flight mission usability of the modified code is - in this context, at this point - not at issue, and not relevant.

Further testing to verify the correctness of individual implementations for the purpose of deployment to those implementations' users' missions is certainly indicated.  But that testing is not the responsibility of CCSDS.

That is, it definitely is the responsibility of CCSDS to perform interoperability test.  But "interoperability testing" in the CCSDS context means a very specific thing.  It means the sort of testing that I have been describing.  Nothing more.

The question of who does the additional testing that may be needed before a user is willing to deploy code on a mission is a very reasonable and important question.  But the answer is not CCSDS.  It is somebody else, e.g., the user's and/or implementer's space agency.

The interoperability testing that the CCSDS CFDP Revisions Working Group is required to perform is limited to the testing that is needed to verify the clarity and completeness of the revisions to the specification.  Nothing more.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 1:57 PM
To: Burleigh, Scott C (312B); Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>; CCSDS All
Cc: sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>; sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>
Subject: Re: [Cesg-all] CFDPv1 status update

Scott -

Back to my question: In the round of updates being discussed in this thread - has any CFDP codebase (NASA or someone else) been modified?

Part of the CCSDS protocol standard publication process is a requirement to perform interoperability testing - if members of the cognizant WG who developed the standard don't interoperability test then who does?  I agree that interoperability test costing is not always a CCSDS-only funded effort - the stakeholders could potentially fund interoperability testing - but to my knowledge it is always the responsible CCSDS WG that either directly conducts, or oversees the testing by an independent contractor, so that the WG can complete the CCSDS-mandated interoperability testing.

Your statement: The only thing the WG is responsible for testing, I believe, is the revisions to the specification, and the way in which we test those specification revisions is to ensure that independent implementations of the revised specification text can interoperate.

The above is interoperability testing, correct?  Whose two independent implementations would be utilized to verify proper interoperability testing?

Thanks.

Kevin

From: <Burleigh>, "Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>
Date: Monday, July 13, 2015 2:29 PM
To: Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Scott, Keith L (9730-Affiliate)" <kscott at mitre.org<mailto:kscott at mitre.org>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: RE: [Cesg-all] CFDPv1 status update

Kevin, I certainly agree that revised implementations need to be regression-tested for correct operation to protect users.  But I don't agree that this implementation regression testing is the responsibility of the CCSDS CFDP Revisions Working Group.

The only thing the WG is responsible for testing, I believe, is the revisions to the specification, and the way in which we test those specification revisions is to ensure that independent implementations of the revised specification text can interoperate.

My understanding is that we do not test software in CCSDS; we only test specifications, by exercising code that was developed from those specifications.  Somebody else needs to pay for the testing of the software itself.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 1:21 PM
To: Burleigh, Scott C (312B); Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>; CCSDS All
Cc: sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>; sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>
Subject: Re: [Cesg-all] CFDPv1 status update

Scott -

Your statement below is: "The purpose of this interoperability testing is not to validate the correct operation of protocol implementations.  It is rather to validate the clarity and completeness of the protocol specification - demonstrating that the document provided sufficient guidance to enable independent implementers to develop software that interoperates."

So, I asked what I was missing and perhaps this is the nugget.  Let me ask this: In the round of updates being discussed in this thread - has any CFDP codebase (NASA or someone else) been modified?  If it's just a specification (documentation) exercise I would agree (FWIW) with your reasoning.  If any code has been changed then I would contend that generally requires retesting to protect the stakeholders/users.

Kevin

From: <Burleigh>, "Scott C (312B)" <scott.c.burleigh at jpl.nasa.gov<mailto:scott.c.burleigh at jpl.nasa.gov>>
Date: Monday, July 13, 2015 2:12 PM
To: Kevin Gifford <Kevin.Gifford at colorado.edu<mailto:Kevin.Gifford at colorado.edu>>, "Scott, Keith L (9730-Affiliate)" <kscott at mitre.org<mailto:kscott at mitre.org>>, "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Cc: "sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>" <sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>>, "sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>" <sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>>
Subject: RE: [Cesg-all] CFDPv1 status update

Kevin, FWIW, I disagree.  I don't think you have stated here what the "clear need" for repeating the interoperability testing of all parts of CFDP might be.

The purpose of this interoperability testing is not to validate the correct operation of protocol implementations.  It is rather to validate the clarity and completeness of the protocol specification - demonstrating that the document provided sufficient guidance to enable independent implementers to develop software that interoperates.

The clarity and completeness of the original, unmodified portions of the specification was validated many years ago, in the extensive interoperability testing that preceded publication of the CFDP Blue Book, and it remains valid because those portions of the specification have not been revised.

What has not yet been validated is the revised text of the specification, and that is what requires interoperability testing now.  I don't think additional testing beyond what is required to prove out the specification can be justified.

Scott

From: Kevin K Gifford [mailto:kevin.gifford at Colorado.EDU]
Sent: Monday, July 13, 2015 9:55 AM
To: Scott, Keith L (9730-Affiliate); Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>; CCSDS All
Cc: Burleigh, Scott C (312B); sis-cfdp at mailman.ccsds.org<mailto:sis-cfdp at mailman.ccsds.org>; sis-dtn at mailman.ccsds.org<mailto:sis-dtn at mailman.ccsds.org>
Subject: Re: [Cesg-all] CFDPv1 status update

Hi Keith -

>From a pedantic test engineering standpoint, I'd strongly suggest regression testing the entire CFDP protocol suite.

Certainly, I am not in-the-loop for the detailed technical protocol updates, but unless the updates were very minor, I see no engineering reason for not testing/retesting/regression-testing the entire updated (and perhaps substantially roto-tilled) codebase.  If the decision is a funds-available business decision, and the updates are very minor, then perhaps with explicit justification the additional testing could be either reasonably (defendable by some reviewed document) waived or possibly postponed.

I understand this increases the testing burden and necessitates (most likely) the need for additional resources.  But if CFDP is used on any current (or future) spacecraft, and especially the ISS in the near-term, then I think not testing the entire updated protocol suite is an engineering mistake (not performing due diligence when there is a clear need).

I just don't see now not retesting the entire codebase can be justified unless the updates are very, very minor.  Am I missing something?

Kevin

From: <Scott>, "Keith L." <kscott at mitre.org<mailto:kscott at mitre.org>>
Date: Monday, July 13, 2015 10:34 AM
To: "Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>" <Secretariat at mailman.ccsds.org<mailto:Secretariat at mailman.ccsds.org>>, CCSDS All <cesg-all at mailman.ccsds.org<mailto:cesg-all at mailman.ccsds.org>>
Subject: [Cesg-all] CFDPv1 status update

All,

The CFDPv1 book completed agency review<http://public.ccsds.org/sites/cwe/rids/Lists/CCSDS%207270P41/Overview.aspx>, and the changes necessitated by the RIDs were minor enough to not warrant another agency review and have been resolved.  The attached book with resolutions, and RID resolution spreadsheet, are for your information.

The working group is going to interoperability test the NEW parts of the protocol only - and that work is under way.  When the interoperability testing is complete SIS will issue a resolution to submit the book for final CESG Poll and then CMC Poll; that resolution will be accompanied by the final version of the book (hopefully same as attached) and the interoperability test report for the new elements of the protocol.

As just mentioned, to reduce the interoperability testing burden, we plan to interoperability test only the NEW portions of CFDP (capabilities added by the current update).  When we submit the final document and interoperability test report, SIS will essentially be requesting a waiver on testing the remaining parts of the protocol.  If anyone knows now that they'll have an issue with this approach, could you please let me know?

V/r,

-keith


From: Scott Burleigh
Date: Monday, July 13, 2015 at 12:16 PM
To: "Scott, Keith L."
Subject: RE: We're ready to go with a resolution to publish the CFDPv1 Blue Book, right?

I think we can assert that none of the technical changes introduced since the agency review affect protocol functionality profoundly enough to warrant the expense of an additional agency review; all of the changes are minor details.  That's going to be a judgment call for CESG, I guess, but I think it's a defensible assertion.

Scott

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/sis-dtn/attachments/20150713/86adf6e3/attachment.html>