[cssm] Non-validated EM data delivery via TGFT

Thu Aug 5 20:00:55 UTC 2021

All ---
The (proposed) delivery of non-validated radiometric (RM) via Terrestrial Generic File Transfer (TGFT) raises issues that affect the Space Communications Cross Support Architecture Requirements Document (SCCS-ARD) and the Functional Resource Model (FRM). In a nutshell, I think there is some missing "glue" specification that automates the "pushing" of the NonVal RM data file through TGFT.

I'll start with an overview of the situation. Interestingly, the NonVal RM "service" does NOT correspond either of the IOAG Service Catalog's "Radio Metric" services (Validated Data RM service and Raw Data RM service). The Validated Data RM service delivers post-pass (off-line) Tracking Data Message (TDM) files containing RM data that has been validated through a provider-specific, presumably-operator-supervised process. The Raw Data RM service provides TM data samples "as soon as they are received/built by a Ground Tracking Asset" (aka in "real time") - typically, every few seconds. The Raw Data RM service is realized by the CSTS Tracking Data Service (CCSDS 922.2).

The NonVal RM service is intended to meet the requirement for what (according to ESA members of the CSTSWG assert} "ESA actually does" - collect raw RM data into TDM-formatted files that are generated over consideably longer intervals than the TD-CSTS samples - e.g., 20 minutes - and send them via a file transfer protocol (as opposed to  the messaging protocol that underlies CSTS service including TD-CSTS). Also, in addition to the tracking data types that TD-CSTS transfers, the NonVal RM service transfers meteorological, tropospheric, and slant total electron count (STEC) data.

As of this time, a candidate Non-validated Radiometric Data Collection functional resource (FR) has been defined in the FRM and an associated set of parameters, events, and directives (PEDs) have been defined for registration in the SANA Functional Resource Registry. The NonVal RM Data Collection FR includes the functions of acquiring the RM data, forming the resultant TDM files, and placing those files is an associated Non-validated RM Data Store. However, for reasons that will be explained shortly, the FRM does *not* address how those files are subsequently transferred their end-recipient, instead simply stating that the files are transferred by bilaterally-defined means.

The SCCS-ARD, on the other hand, asserts a standard "interface binding signature" in which the transfer of NonVal RM TDMs occurs over TGFT. Indeed this is the desired standardized approach, but not all the standard pieces are in place to implement it unambiguously introperable.

The issue is in the specification (or, rather, lack thereof) of the process by which the TDM files created by the NonVal RM Data Collection FR are transformed into the TGFT XFDU Packages that are required by TGFT, and subsequently "pushed" through TGFT to the recipient.

The TGFT XFDU Package is a refined (slimmed-down) and (very slightly) extended version of the CCSDS XFDU Package, which was designed for archiving and retrieval of space data, and supports merging of data from different archives in different formats. Simply put, the CCSDS XFDU has tons of "features" that are not applicable to the simpler type of file transfer that was envisioned for TGFT. Nevertheless, it was selected as the file packaging mechanism for TGFT in order to reuse an existing CCSDS standard and "not re-invent the wheel" (although in this case the "wheel" is a military-grade puncture-proof, fire-proof, bomb-proof wheel that we are putting onto a bicycle).

The TGFT XFDU Package is a ZIP or TAR file that contains (a) one or more "payload" data files, (b) zero or more metadata files, and (c) an XML formatted Manifest file that defines the relationships among the enclosed payload data file and metadata files, and optionally relationships to external metadata files (e.g., externally-defined XML schema files). If the application's "payload" data file format already includes all necessary metadata, then the XFDU Package reduces to just the payload data file itself and a simple Manifest file. For Non-Val RM data, the payload data file is the TDM file. Among the things that have not yet been specified are whether or not there need to be any accompanying (or referenced) metadata files, and if so, the contents and/or locations of those metadata files. Can a provider pack more than one TDM into an XFDU Package, or should a new XFDU Package be generated for each TDM file (i.e., are there any timeliness requirements for this less-than-real-time service)?

TGFT is a push-only "protocol",  so once a NonVal TDM file is generated, the resulting TGFT XFDU Package must be created and pushed through the TGFT Sender to the TGFT Receiver associated with the target application. So the "service request"(so to speak) to the TGFT Sender entity has the arguments TGFT XFDU Package and the URL of the "in-tray" of that target TGFT Receiver. On the receiving side, the target application somehow "knows" to poll the in-tray for files being sent to it. The bottom line implication of this is that the XFDU Package file name has to convey enough information to allow the target application to recognize it. So other missing pieces of the puzzle are the naming convention that translates the name of the TDM file to the more-general file-naming conventions already specified for TGFT, and the method (e.g., managed parameters?) by which the TGFT Sender knows the URL of the "in-tray" of the target TGFT Receiver.

TGFT uses HTTP over TLS, also known as HTTPS. By my (almost nonexistent) knowledge of HTTP, the "client" must open a TCP connection and then send an HTTP request, at which point the server uses the same TCP connection to return the corresponding HTTP response. Presumably the TGFT Sender (in the ESLT) is the "client" in this case. I don't know how this paradigm is intended to be used in a push-only scenario. Does the HTTP request contain the XFDU Package itself (the POST method), or something else (such as just the URL of the XFDU Package that the TGFT Receiver subsequently uses to pull the file)? What does the TGFT Receiver send back as the corresponding HTTP response? It's not at all obvious to me how this would work, but perhaps to someone who has more than my Wikipedia-level understanding of HTTP it is obvious. In any case, this, too, should be included in the missing specification.

Unless and until we have "standard" answers to these questions, I am reluctant to state (as the SCCS-ARD currently does) that automated transfer of nonvalidated RM data TDM files over TGFT is a currently-available capability.

I would be very happy if someone already has the answers to all these questions and all that would be required would be to document them somewhere (where to document them would be a follow-on question).

Comments?

Best regards,
John

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/smwg/attachments/20210805/a00765a5/attachment.htm>