[Smwg] TGFT yellow book

[John Pietras]

Dear all,
I have reviewed the comments that Claudia submitted on 22 December and the comments that Wes submitted on 3 January, and I have updated the TGFT XFDU Package in response to those comments that applied to the TGFT XFDU schema, the user/provider vs. sender/receiver terminology, and typo cleanup.

The XFDU Package - xfdu_for_tgft-package.zip-2018-004T22-13-00Z - is a zipped file that conforms to the TGFT XFDU requirements (or at least it is intended to do so) that has been posted to the CWE at URL
https://cwe.ccsds.org/css/docs/CSS-SM/CWE%20Private%20-%20Beta/Book%20Production/Blue/Terrestrial%20Generic%20File%20Transfer/White%20Book/Drafts/xfdu_for_tgft-package.zip-2018-004T22-13-00Z.

The XFDU Package contain the following files:

1.       xfdu_for_tgft-package-manifest.xfdu. The manifest XFDU document for the package. Compared to the manifest file in the  previous version of the Package, the file name now conforms to the character set constraints for TGFT files. Other changes are identified in the detailed comments in the remainder of this email.

2.       XFDUschema.xsd. The original CCSDS XFDU schema file. This is unchanged from the previous version of the Package.

3.       TGFTXFDUschema.xsd. The schema has been updated as described in the detailed comments bellow.

4.       Terrestrial Generic File Transfer 927x1w0.02-JVP-rev1-180103.doc. This is the most recent draft of the TGFT White Book that I have. It consists of Colin's version 0.02 white book with my comments and suggested re-organization of the XFDU material, PLUS changes that resulted from Claudia's and Wes' comments. Because the most important parts of the XFDU in TGFT Tech Note have been transferred to the TGFT White Book, the TGFT White Book is now the authoritative document on the TGFT XFDU schema, and the Tech Note will no longer be updated.

5.       tgft_xfdu_manifest_tdm_example-180103.xml. This is an update of the TDM manifest example file that appeared in previous versions of the Package. Compared to the manifest file in the  previous version of the Package, the file name now conforms to the character set constraints for TGFT files. Other changes are identified in the detailed comments in the remainder of this email. Note that this file has the .xml extension because it is not the manifest file for this XFDU Package (i.e., it is simply a payload data file here). If it were the manifest file for its own XFDU Package, it would have the .xfdu extension.

Below are copies of Claudia's and Wes' emails. Point-by-point responses are interleaved into those email copies in red.

Unfortunately, it looks like Colin won't be back at work until the day of our next WebEx, so he will not have had the chance to look at and respond to these inputs at that meeting.

Best regards,
John

From: SMWG [mailto:smwg-bounces at mailman.ccsds.org] On Behalf Of Ciocirlan Claudia
Sent: Friday, December 22, 2017 9:39 AM
To: Barkley, Erik J (3970); Colin.Haddow at esa.int; CCSDS SMWG ML (smwg at mailman.ccsds.org)
Subject: [Smwg] TGFT yellow book

Dear all,

I know I am early on planning but I would like to propose you a first draft of the TGFT yellow book. It will be simpler to discuss over it in January.

*        I know it is not what we agreed on, but it is only a proposal for the cross validation. Maybe by seeing it on paper it makes more sense than explaining. This can easily be changed.

I have a couple of questions that my contractor mentioned and as I do not have the answer I am addressing his questions and remarks to you. As the comments address to various books and John example I rather prefer laying them in an email, but I can integrate them in the documents is needed.

Could it be possible to add this on the agenda of the 16th of January?

TGFTXFDUschema.xsd
This schema is defined but it is not used in the example manifest sent by John. The standard XFDUschema is used instead. Is there any reason for this?
The reason is that I wanted to make sure that the TGFT XFDU manifests were indeed still conformant to the original XFDU schema (which they were and are) and I forgot to reset the referenced schema fil to "TGFTXFDUschema.xsd". This change has now been made in the two manifest files and in Annex E of the TGFT White Book in the xfdu_for_tgft-package file.

However, the TGFTXFDU schema still resides in the urn:ccsds:schema:xfdu:1 namespace. I think that my original motivation for leaving it in that namespace was that since the TGFT XFDU schema is a subset of the XFDU schema, it could reside in the same namespace, but that may have been an incorrect assumption. Also, to truly be a "proper" subset of the XFDU schema, the TGFT XFDU schema should be formally derived from the XFDU schema with restrictions (which we can do). We should discuss the namespace of the TGFT XFDU schema.


The value of this attribute of the volumeInfo element is declared fixed in the specification (section 4.2.5.4.1) but not in the schema. It could be. The specificationVersion element shall have the value "1.0".
The value of the specificationVersion element has been fixed to "1.0" in the TGFTXFDUschema.xsd file.

The specification (section 4.2.7.2.2) requires a metadataReference element in a metadataObject, however this is declared in the schema with minOccurs="0".The metadataObject element shall contain a metadataReference element that identifies the location of the corresponding metadata file.
The TGFTXFDUschema has been changed to have minOccurs = 1 for the metadataReference element, and in figure F-12 of the draft White Book that is contained within the Package.

Also, because the metadataSection is present only if there is at least one metadataObject, the minOccurs for the metadataObject must be 1 instead of 0 (zero). This has been changed in the TGFTXFDUschema and in figure F-12 of the draft White Book that is contained within the Package. It has also been clarified in 4.2.4 (c) of the draft White Book.

The specification (section 4.2.8.2.2) requires a single byteStream element in a dataObject, however the schema specifies an unbounded number. The dataObject element shall contain one byteStream element.
The TGFTXFDUschema has been changed to have maxOccurs = 1 for the byteStream element. (Curiously, the XMLSpy diagram for the dataObjectSectionType was already corrected in the draft White Book - I must have made the change in the copy of the schema file that I used to generate the diagrams but failed to include it in the schema file that I distributed.)

Standard white book 927x1w0.02
PackageHeader ID (section 4.2.5.1.a)
The remark below relates to the use of "shall" in the specification. Shall has a strong meaning as a conformance requirement. However in the cases below, this requirement may be changed by the TGFT-using services, and thus not respected. A different wording without shall would be preferable.
The ID attribute of the packageHeader element shall have the default value of "pkgHdr". However, TGFT-using service may define their own values for the ID attribute.
The intent of this statement was that if the creator of the Package does not have their own value for the ID attribute, then the value should be set to "pkgHdr". In general terms, I believe that is what is meant by "default value" - it is the value unless specifically overridden.

However, the XML Schema definition of "default" has particular semantics that prohibit assigning a default value to a required attribute such as ID. So instead of using "default" and risking confusion, the requirement in 4.2.5.1 (a) has been rephrased in the draft White Book to read "The ID attribute of the packageHeader element shall have either (1) a value that is relevant to the TGFT-using application or  (2) the value "pkgHdr".

SequenceInformation (section 4.2.5.5.2)
This is similar to the previous remark, with the use of a shall. The sequencePosition attribute shall contain the numerical position of the XFDU in the XFDU sequence. The semantics of this attribute are deferred to the specification of the service.
The sequenceSize attribute shall contain size of the XFDU sequence to which this SFDU belongs. The semantics of this attribute are deferred to the specification of the service.
In both cases, "shall" has been changed to "is intended to be used to".


MetadataObject
A requirement in section 4.5.3 looks inconsistent with the rest of the document. The TGFT XFDU package shall contain one enclosed metadata files for every metadataObject element in the metadataSection of the XFDU manifest. A metadata file may not be included in the XFDU package (cf for example section 4.2.7.3.1.c.2).
Good catch! 4.5.3 has been changed to read "The TGFT XFDU package shall contain one enclosed metadata file for every metadataObject element in the metadataSection of the XFDU manifest for which the metadataReference element has an href value that begins "file:"."


Naming rules
Inconsistencies exist in the naming rules (section 4.5.4.*) and their example usages (Annex E). In the annex the XFDU zip file is named dss_25_validated_tdm_xfdu_package-2017- 058T23-15-46Z.zip. When the timestamp part defined in 4.5.4.2 is removed, the servicespecific part should remain as defined in 4.5.4. The service-specific part should then be dss_25_validated_tdm_xfdu_package- and not dss_25_validated_tdm_xfdu_package as shown in the example. Nothing is said in 4.5.4 about adding a dash in between.
Thanks for pointing out these inconsistencies. The timestamped name of the XFDU package should be "dss_25_validated_tdm_xfdu_package.zip-2017-058T23-15-46Z" so that when the timestamp part is discarded the original file name remains. I agree that Colin's rules for combining the service-specific part and the timestamp part did not explicitly include the dash between them in 4.5.4, but his example in  3.2.1 did include the dash (file_name.file_type-YYYY-DDDThh:mm:ssZ).

In the consistency and consolidation changes that I proposed to Colin in November, I proposed to explicitly add the dash as part of the syntax of the transferred XFDU Package name. The new name syntax is specified in section 3.2.1.1 of the updated draft TGFT White Book that is contained in the zipped package that I just uploaded to the CWE.

Also, the name of the example  transferred XFDU Package Annex F has been changed in the draft TGFT White Book.


Manifest file
Few rules apply to the naming of the XFDU manifest file. However, looking at the examples, I wonder if one rule could be added concerning the location of this file. I expected it to be found at the top level in the archive, however it has been set in the first level directory.
I don't understand this question. Let's talk about it at the January WebEx.

Annex E
The name of the file (dss_25_validated_tdm-2017-058T19-35-24Z.xml) does not conform to the naming rule in section 3.2.1. The upper case letters are forbidden. The Zulu time exception described in the specification only concerns the name of the archive itself, not the files inside it.
The reference to the file (href attribute) does not conform either to the specification as defined in section 4.2.8.4.1.c. The Zulu time part of the archive name should be removed from the directory part of the href. The href attribute shall contain the URL of the file, which shall be of the form "file:"+<XFDU Package file name service-specific part> + "/" + <payload file name (with extension>
Colin and I discussed (in October or November 2016, I believe) the applicability (or lack thereof) of the file-naming constraints and agreed that they should NOT apply to the payload and metadata file names because (a) TGFT sees only the name of the XFDU Package file and (b) the applications/domains that generate the payload data and metadata files that are contained within the XFDU Package may have their own file-naming conventions. The exemption of the payloads data file and metadata file names from the file-naming constraints was documented in sections 4.3.2 and 4.4.2 of the version 0.02 TGFT White Book but unfortunately this was not correctly reflected in section  3.2.1. It is one of the items that I cleaned up in the set of suggested changes that I made to Colin in November - see the revised section 3.2.1 of the draft TGFT White Book that is included in the XFDU Package described above.

The file with the name "dss_25_validated_tdm-2017-058T19-35-24Z.xml" is a payload data (and thus exempt from the TGFT Manifest/Package naming conventions) and the "2017-058T19-35-24Z" portion of the filename refers to the time at which the tracking data stopped being acquired.  The use of the timestamp in this  case is presumed to be part of the (hypothetical) Validated Radiometric Data cross support service file naming convention. However, I can now see that the use of the same timestamp format as that of the transferred TGFT XFDU Package name can be confusing. Therefore I have changed the name in the example to use a time/data stamp that follows  the XSD datetime format (CCYY-MM-DDThh:mm:ssZ). Thus the filename in the example has been changed to "dss_25_validated_tdm-2017-02-07T19-35-24Z.xml" to emphasize that this filename data/timestamp is not related to the TGFT XFDU Package file name conventions.

TGFT/XFDU manifest
XFDU version
The version attribute is declared while it should not be (cf section 4.2.3.b). The optional version attribute shall be absent when the XFDU schema defined in Issue 1 (September 2008) of reference [9] is used as the schema for TGFT XFDUs.
Thanks for catching that. The version attributes have been deleted from both the manifest for the TGFT documentation package and the example TDM manifest.


PackageHeader ID
This attribute is set to "PkgHdr" instead of "pckHdr" (cf section 4.2.5.1.a). The ID attribute of the packageHeader element shall have the default value of "pkgHdr". However, TGFT-using service may define their own values for the ID attribute.
As described above under the Package Header ID section, the intention was that an application could set its own value for the ID, or it could "default" to "pkgHdr". The real driver here is that since the ID field is mandatory in this CCSDS-standard ID attribute, "something" must be supplied and the value "pkgHdr" could be used if the application had no specific use for this attributeTechnically, "PkgHdr" can be viewed as an application-specific use for this attribute and is valid.


File naming
The file names include upper case letters which are forbidden by the specification (cf section 3.2.1). This remark may lead to reconsider this rule. One of the  files in the example package is the XSD schema. Its name is TGFTXFDUschema.xsd, and this name could/should be included in the xml files that conform to the schema. If we rename the file to tgft_xfdu_schema.xsd for example, to conform to the specification, then the XML file will loose its reference to its model.
As noted above, the file naming conventions apply to only the manifest and package files names, TGFTXFDUschema.xsd is a payload data file and is therefore not required to follow the convention. Similarly, the XFDUschema.xsd file is also a payload data file, and it is a good example of why payload data and metadata file names are exempt from the convention, since the name of the file was assigned by the CCSDS Navigation WG.

Similar issues are very likely to occur in the TGFT use cases, as the example in Annex E shows. Are these issues been identified? Has a proper handling procedure been suggested?
I think that we will avoid similar issues now that we have made the file-naming convention apply to only the manifest and package file names.

Size
The size attributes provided in the example do not match the actual size of the files as I see them on my PC (Windows). Is this just an update issue of the values, or is there something more?
I forgot to update the sizes when I modified the contents of the files within the zipped folder. That has been corrected in the latest manifest.

Wish you all  Merry Christmas and a veyyrryyyy happpyyyy new year!

Best regards,
Claudia

Claudia Ciocirlan CNES/DNO/OP/SSO
Mission Operations Ground Segment
phone : +33 (0)5 61 28 17 11
email : claudia.ciocirlan at cnes.fr<mailto:claudia.ciocirlan at cnes.fr>
P Save a tree ! Think before printing






________________________________

From: Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.] [mailto:wesley.m.eddy at nasa.gov]
Sent: Wednesday, January 03, 2018 3:52 PM
To: John Pietras
Cc: Barkley, Erik J (JPL-3970)[Jet Propulsion Laboratory]; Colin.Haddow at esa.int
Subject: RE: Updated TGFT White Book on CWE

Hi John, I took a look at your draft, and here are a few comments, questions, and typo catches on the TGFT book you posted.  I copied Erik and Colin, but don't know if this is something the rest of the working group would normally be interested in, or if comments are typically just unicast to the book editors.

First just a few easy typos:

  *   page 14 (or 1-2), in list item b) bullet q. should "vent" be "event"?

     *   in list item c) should "Post Mission" be "Post-mission"?

  *   page 16 (or 1-4), in section 1.3.2, first paragraph, last sentence, "seem" should be "seen"
  *   page 17 (or 1-5), in 2nd paragraph, "underlying TFGT" should be "underlying "TGFT"?
  *   page 19 (or 1-7), in item h) "containg" should be "containing"
  *   page 25 (or 3-1) in the first paragraph of 3.1 "Senderto" should be "Sender to"
  *   page 28, the section title for 3.2 is jumbled somehow
  *   bad reference in 3.2.1.1 part a)
  *   page 31 (or 3-7), "deliveredwith" is missing a space and "paload" should be "payload"
  *   page 34, in 4.2.5.2 "volumenInfo" looks like it has an extra n?
All of the typos have been corrected


More substantial thoughts/comments/questions:


  1.  On page 27 (or 3-3) in the 2nd paragraph, I think the service provider/user terminology should more correctly be file sender/receiver (as the terminology is discussed earlier in the document).
There were several user/provider, source/recipient, etc., roles described in the White Book, and they were not used consistently. I took a stab at reworking the relationships and proposed them to Colin in November. The resulting "homogenization" of those terms is present in the draft TGFT White Book that is contained in the latest Package on the CWE. Without repeating all of the new proposed definitions, in light of those (proposed) new definitions, the sentence to which you are referring has changed from
"The PROPFIND method thus enables the service provider to retrieve the properties of the directory hierarchy of the service user system and can thus check that the file has successfully been transferred." to
"The PROPFIND method thus enables the file source to retrieve the properties of the directory hierarchy of the TGFT Receiver and can thus check that the file has successfully been transferred."
 where "file source" refers to the entity (e.g., application, operator) that uses TGFT to send the file, and "TGFT Receiver" is the host computer that receives the file.

Note that in this particular case, however, the use of PROPFIND is under question and this whole subsection may disappear.


This is the extent to which I can respond to your comments. Colin and/or the rest of the working group will have to address your remaining comments.





  1.  In section 1.8, why are TLS 1.0 and 1.1 references provided in addition to TLS 1.2?  It wasn't clear if there is some technical or practical reason.  Both are obsoleted, and TLS 1.3 is going to be standardized soon, so it's not clear why anything more than just 1.2 would make sense.


  1.  Is TGFT only intended to work with HTTP 1.1 and not HTTP/2?  maybe this is related to relying on WebDAV, since it's unclear if people are using these together yet in industry?  Maybe this is something to bookmark for future work, but not worry about now, since the software tooling available predominately supports 1.1 and it isn't really obsolete yet?


  1.  It probably doesn't matter much in practice as TGFT is envisioned to be used, but there may be some friction between use of "file" concepts in TGFT versus more generic concept of "resources" in HTTP.  In the HTTP server, the resources are often stored and managed as files, but definitely not always (e.g. in the case of REST applications, etc).  I doubt this is a real issue for us at the moment, but maybe just a terminology question to think about, if perhaps there is some use case where the XFDUs and contents are processed straight to/from database entries or something.  That would be more like a typical REST application, and I guess TGFT is using HTTP and WebDAV for transport of XFDUs as files *only*, and not really intended to fit into a REST-like architecture?


  1.  For cross-system compatibility, TGFT limits the valid characters that can be used in the filename, but I didn't notice a limit on the length of a filename?  It seems like there should be a limit on the number of characters for similar compatibility reasons.


  1.  Should there be a limit to the size of files and/or XFDUs?  Maybe this is something for appendix A, in addition to the data volume already there?


  1.  In 3.3, there is a suggestion to append a hash value within the filename with a colon separator, but the colon is illegal in filenames in some systems, and adding it could make the filename too long?


  1.  Hash values or signatures can just be provided separately in files on the side, if needed.  This is how it's done normally for large downloads on the Internet like Linux installation images, signed RPM files, etc.  I don't think doing anything fancier (like embedding in the filename) adds value, and probably creates complexity.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/smwg/attachments/20180104/2bf33a09/attachment.html>