[Smwg] updated draft TGFT on CWE

Wed May 9 19:34:13 UTC 2018

John,

I know we discussed this a bit at the teleconference yesterday but I have also provided some comments which are listed below.

Best regards,

-Erik

From: SMWG <smwg-bounces at mailman.ccsds.org> On Behalf Of John Pietras
Sent: Tuesday, May 1, 2018 11:41 AM
To: CCSDS SMWG ML (smwg at mailman.ccsds.org) <smwg at mailman.ccsds.org>
Subject: [Smwg] updated draft TGFT on CWE

CSSMWG colleagues ---
At the Gaithersburg meeting, we discussed several items raised by Claudia that she and her team discovered during TGFT prototyping. I took that action to update the TGFT White Book. I have performed the update (to v0.4) and posted it to the CWE at URL
https://cwe.ccsds.org/css/docs/CSS-SM/CWE%20Private/Book%20Production/Blue/Terrestrial%20Generic%20File%20Transfer/White%20Book/Drafts/927x01w0.4%20-%20Terrestrial%20Generic%20File%20Transfer-180501.doc

What follows is my reconstruction of the major discussion points and how they have led to the changes in the updated White Book. Please review these point to see if they accurately reflect your recollection of the decisions made at the meeting.

However, before going into the recap of the meeting, I'd like to raise a concern that occurred to me while I was making the updates. According to the White Book, each Agency has a single "in-tray" for all files received from another Agency. Since TGFT is expected to be used for multiple applications/services concurrently, this means that there may be files for different applications residing in the in-tray. How does the Receiver distinguish, for example a Return File Service file (contain spacecraft telemetry) from a validated Radiometric Data file? Because there are no file naming conventions that could be used to differentiate between the applications that created the files (theoretically, TGFT would be perfectly happy if all files were simply named "myData-<timestamp>.zip") I had put into the TGFT XFDU definition that the packageType attribute of the informationPackageMapType (see F6.3.1 of the White Book) is required to be present and used to distinguish the applications. This requires that any "sorting" application of the receiving end burrow down into the XFDU Manifest to read this attribute to determine how to process the XFDU Package. To my knowledge we've never discussed this. Is it acceptable, and is this the method for file differentiation that is being used in the prototypes? If it is acceptable, then I still think that we need some sort of SANA Registry to register the different standard (string) values of  packageType. If it's not a good way to do this (e.g., because it requires looking into the Manifest file within the compressed XFDU Package file) then how should we handle it?

eb> Certainly the idea of registering the different file types has always been part of the TGFT concept as best as I can recall. I realize that you are also looking at the identification within the XFDU package. I believe that as a result of the teleconference yesterday Colin as a subsequent action item for further consideration in this regard.

Best regards,
John

1.       The dataObjectPointer type was erroneously called "dataPointerObject" in multiple instances in the text of the TGFT book. This has been fixed in the latest update.

2.       The standard CCSDS XFDU provides two schema extension mechanisms as part of the environmentInfo element of the packageHeaderType: the xmlData element containing unstructured XML, and the extension element by which the XFDU schema can be extended through the use of an XML type. The TGFT XFDU formally specifies that the extension element is the specified mechanism for extensions to the schema for the base TGFT service, and that such extensions must be done through the TgftXfduExtensionType complex schema type. However, the TGFT XFDU specification does not forbid any user-application-specific use of the (unstructured) xmlData elements. Claudia suggested that the xmlData element be explicitly forbidden to be used in TGFT XFDUs.

In Gaithersburg, we discussed the difference between what additional metadata might be standard for *all* TGFT XFDUs and metadata that a TGFT-using application might need to add to the XFDU for its own application-specific purposes. Examples of extension metadata that is common to all TGFT XFDUs are originator, recipient, and creationDate, where originator and recipient are included in the TgftXfduExtensionType complex schema type because they apply to the whole XFDU Package, and creationDate is in the TgftContentUnitExtensionType because it applies to each individual Content Unit (i.e., payload data file) within the XFDU Package.

We agreed that (a) extension metadata that is common to all TGFT XFDUs (and thus part of the TGFT standard itself) must be contained in the TgftXfduExtensionType or TgftContentUnitExtensionType complex schema types, and (b) TGFT-using applications *may* use xmlData elements for adding application-specific metadata that applies to the XFDU Package as a whole. I agreed to review the description of this distinction in the TGFT book and clarify it if necessary. I have since looked at the relevant sections of the TGFT book and have concluded that it is indeed ambiguous and in need of clarification. As the result:

a)       I've reworded the text to state that the recommended extension mechanism for adding TGFT-using application-specific metadata that applies to the XFDU Package as a whole is through the addition of another environmentInfo element containing an extension element.

b)      I've added a NOTE that says that TGFT-using applications may also use xmlData elements, but any such usage is outside the scope of TGFT and must be documented as part of the specification of the TGFT-using application.
eb> That sounds reasonable to me. I'd like to get Claudia's take on your note as obviously this does not explicitly forbid the use of xmlData element.

3.       We also discussed the inconsistencies in the construction of the XFDU Package file names and the href elements in the XFDU Manifest that point to the files contained in the XFDU Package. As we discussed, I have attempted to fix these inconsistencies by reformulating the XFDU Package file name (as transferred by the TGFT Sender tot the TGFT Receiver) as having three components:

a.       the package folder name part;

b.       the compressed folder file extension type; and

c.       the timestamp part.

The package folder name part is the name of the folder (subdirectory) that contains all of the files contained within the XFDU Package: the manifest file, any payload data files, and any contained metadata files. The compressed folder file extension type is the extension that specifies which compression method (".zip" or ".tar") has been used to compress the folder into a single file for transmission.

We also agreed that the compressed folder file extension type would be at the end of the XFDU Package file name (in the "normal" position for a file name). This, when an XFDU Package "folder" that has the folder name "XXX" is compressed using ZIP, it has an original compressed package file name "XXX.zip". When that file is subsequently transmitted from the TGFT Sender to the TGFT Receiver, the TGFT Sender inserts a timestamp part to form the as-transmitted file named "XXX-<timestamp part>.zip". As we agreed, when the file XXX-<timestamp part>.zip is subsequently unzipped on the TGFT Receiver side, the uncompressed folder is simply name "XXX" - that is, timestamp part is lost. I have reworked the daft White Book in accordance with these understandings.
(eb> well it might be a rather daft undertaking but I will take this as the draft white book :-) )

NOTE - Section 3.3 (Security) of the White Book identifies a security mechanism that includes post-pending a SHA-2 hash value to the file name. However, to my knowledge a final decision on this approach has not been reached, and the following discussion does not address this aspect of the file name. If such a post-pended hash value is added to the standard, the exact syntax should be addressed in section 3.2.1.1 (XFDU Package File Names).
eb>  yes, here again, for the benefit of those not in the teleconference, Colin will be taking a look at this.

4.       The next issue raised by Claudia was in regard to inconsistent naming in the example in Annex E. These errors were the result of my changing the timestamp for the TDM file from CSDS time code B to xsd:datetime format but not propagating the change throughout the example (2017-058 and 2017-02-27 are both 27 Feb 2017). These have been corrected.

5.       This next item is not one that was explicitly initiated by Claudia, but is a result of my looking into the issues that Claudia raised.  The TgftXfduExtensionType complex schema type has three elements, two (originator and recipient) of which represent TGFT metadata that the CSSMWG put forward in the TGFT brainstorming session in Rome (which of course I did not attend) as candidate metadata elements that do not otherwise appear in the CCSDS-standard XFDU, and the third (crossSupportServiceType) is something that I added because it seemed like something useful (note that not all TGFT XFDUs will necessarily be generated by cross support services: there may be Provider CSSS-unique or Mission-unique uses of TGFT. In such XFDUs, the optional crossSupportServiceType would not appear).

All three elements (originator, recipient, and tgftUsingApplicationType) are currently of type String, and otherwise undefined. At what level are originator and recipient defined - Provider CSSS/Mission, individual ground station/Mission facility, etc.? Where are the valid values of originator and recipient defined and/or registered? Do we need to register the values of crossSupportServiceType, in order to ensure that different user applications don't use the same value?  I've flagged these issues as comments in the TGFT White Book.

eb> I am not sure that I follow. In the first paragraph of item 5 the three elements are called out as originator, recipient, and crossSupportServiceType and the second paragraph says that the three elements are originator, recipient, and tgftUsingApplicationType. In any case, it does not seem to me that the cross support service type adds anything? -- especially if the various types of files are already registered in SANA.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/smwg/attachments/20180509/ab40e138/attachment.html>