[Css-csts] Non-validated EM data delivery via TGFT

Sat Aug 21 00:25:44 UTC 2021

John,

On the face of it, it seems rather audacious for the ARD to state that any automated capability of any kind can be stated as "currently-available" (as you have noted "...I am reluctant to state (as the SCCS-ARD currently does) that automated transfer of nonvalidated RM data TDM files over TGFT is a currently-available capability.")   To the best of my knowledge, ARD == Architecture Requirements Document and is not any kind of service catalog.  I think the ARD is supposed to say that if you want to achieve a certain function in standard way these are then the standards that are required, right?

Validated vs non-validated radio metric data service has always been something outside the purview of the CSS Area. I believe these concerns/definition of just what validated or non-validated radio metric data service does are best addressed by the MOIMS NAVWG.   And, to the best of my knowledge they generally do not do so - the TDM is a data format designed to support delivery of tracking/radio-metric data, but it does not address/specify how that data is produced, how it is processed for validation, etc.  Indeed, this is what the TDM book says right at the start of section 2 "...This section provides a high-level overview of the CCSDS recommended Tracking Data Message, a message format designed to facilitate standardized exchange of spacecraft tracking data between space agencies.."

We get a near real-time data transfer service for RM/tracking data with TD-CSTS as this leverages the TDM to supply streaming tracking and it indicates what parameters to put in the TDM segments, But certainly TD-CSTS says nothing about how, for example, ranging data is arrived at in the first place (i.e, it says nothing about sequential tone vs PN ranging, square waves, sine waves, etc.)  And I think this is exactly as it should be - the CCSDS specs for the CSS Area indicate how to transfer and what to transfer but not the service that is producing, in this case, the radio metric data per se.

I fail to see how the situation is any different for TGFT.   It is just files of TDMs packaged in XFDUs, as opposed to TDMs that are streamed in realtime.  In either case, the TDM is produced and then put into some other context (streaming or file) for delivery.  In both the TD-CSTS and TGFT cases, they do not say how to go about running the service.  It is then specific implementations that in fact work out how to meet the interface spec given what that implementation's service does.

Now, if there is to be standard practice for how a particular file type is to be constructed in slightly-TGFT-restricted XFDUs for subsequent transfer via TGFT, then there can certainly be a service spec for that service which just points to TGFT for doing the transfer.  But it can not be the "job" of the TGFT spec to state how it itself is to used in all the various contexts in which CCSDS member agencies choose to put it to use (by definition, "mission impossible").   How many TDMs to pack in a file, etc., is clearly a concern for the "upstream" using service and not a TGFT concern.  The TGFT recommendation includes a SANA registry so that the various filetypes (called package types as more general term in TGFT) are recorded, "known".  So I think your concern re "XFDU Package file name has to convey enough information to allow the target application to recognize it" is already addressed.  Similarly the TGFT recommendation contains service agreement parameters that address your concerns re "...the method (e.g., managed parameters?) by which the TGFT Sender knows the URL of the "in-tray" of the target TGFT Receiver..."   (please see Annex B of the TGFT document).

With regard to HTTP(s) PUT, the TGFT document includes a reference to RFC 2616 which is the spec for HTTP  (see https://datatracker.ietf.org/doc/html/rfc2616 ). I believe you will find the necessary details for the PUT method along with  other details for HTTP in general.    I don't think it is the job of TGFT to explain how HTTP works.  Please keep in mind that CNES and CAST were able to successfully exchange files via the TGFT specifications - this was accomplished quite simply via unix curl command line calls.

Best regards,
-Erik

From: John Pietras <john.pietras at gst.com>
Sent: Thursday, August 5, 2021 13:01
To: CCSDS SMWG ML (smwg at mailman.ccsds.org) <smwg at mailman.ccsds.org>; SEA-SA <sea-sa at mailman.ccsds.org>; CCSDS_CSTSWG (css-csts at mailman.ccsds.org) <css-csts at mailman.ccsds.org>; Wolfgang Hell <wo_._he at t-online.de>
Cc: Barkley, Erik J (US 3970) <erik.j.barkley at jpl.nasa.gov>; Colin.Haddow at esa.int; Holger.Dreihahn at esa.int; Shames, Peter M (US 312B) <peter.m.shames at jpl.nasa.gov>; Pham, Timothy T (US 3300) <timothy.t.pham at jpl.nasa.gov>; matteo.renesto at telespazio.de; Wolfgang Hell <wo_._he at t-online.de>
Subject: [EXTERNAL] Non-validated EM data delivery via TGFT

All ---
The (proposed) delivery of non-validated radiometric (RM) via Terrestrial Generic File Transfer (TGFT) raises issues that affect the Space Communications Cross Support Architecture Requirements Document (SCCS-ARD) and the Functional Resource Model (FRM). In a nutshell, I think there is some missing "glue" specification that automates the "pushing" of the NonVal RM data file through TGFT.

I'll start with an overview of the situation. Interestingly, the NonVal RM "service" does NOT correspond either of the IOAG Service Catalog's "Radio Metric" services (Validated Data RM service and Raw Data RM service). The Validated Data RM service delivers post-pass (off-line) Tracking Data Message (TDM) files containing RM data that has been validated through a provider-specific, presumably-operator-supervised process. The Raw Data RM service provides TM data samples "as soon as they are received/built by a Ground Tracking Asset" (aka in "real time") - typically, every few seconds. The Raw Data RM service is realized by the CSTS Tracking Data Service (CCSDS 922.2).

The NonVal RM service is intended to meet the requirement for what (according to ESA members of the CSTSWG assert} "ESA actually does" - collect raw RM data into TDM-formatted files that are generated over consideably longer intervals than the TD-CSTS samples - e.g., 20 minutes - and send them via a file transfer protocol (as opposed to  the messaging protocol that underlies CSTS service including TD-CSTS). Also, in addition to the tracking data types that TD-CSTS transfers, the NonVal RM service transfers meteorological, tropospheric, and slant total electron count (STEC) data.

As of this time, a candidate Non-validated Radiometric Data Collection functional resource (FR) has been defined in the FRM and an associated set of parameters, events, and directives (PEDs) have been defined for registration in the SANA Functional Resource Registry. The NonVal RM Data Collection FR includes the functions of acquiring the RM data, forming the resultant TDM files, and placing those files is an associated Non-validated RM Data Store. However, for reasons that will be explained shortly, the FRM does *not* address how those files are subsequently transferred their end-recipient, instead simply stating that the files are transferred by bilaterally-defined means.

The SCCS-ARD, on the other hand, asserts a standard "interface binding signature" in which the transfer of NonVal RM TDMs occurs over TGFT. Indeed this is the desired standardized approach, but not all the standard pieces are in place to implement it unambiguously introperable.

The issue is in the specification (or, rather, lack thereof) of the process by which the TDM files created by the NonVal RM Data Collection FR are transformed into the TGFT XFDU Packages that are required by TGFT, and subsequently "pushed" through TGFT to the recipient.

The TGFT XFDU Package is a refined (slimmed-down) and (very slightly) extended version of the CCSDS XFDU Package, which was designed for archiving and retrieval of space data, and supports merging of data from different archives in different formats. Simply put, the CCSDS XFDU has tons of "features" that are not applicable to the simpler type of file transfer that was envisioned for TGFT. Nevertheless, it was selected as the file packaging mechanism for TGFT in order to reuse an existing CCSDS standard and "not re-invent the wheel" (although in this case the "wheel" is a military-grade puncture-proof, fire-proof, bomb-proof wheel that we are putting onto a bicycle).

The TGFT XFDU Package is a ZIP or TAR file that contains (a) one or more "payload" data files, (b) zero or more metadata files, and (c) an XML formatted Manifest file that defines the relationships among the enclosed payload data file and metadata files, and optionally relationships to external metadata files (e.g., externally-defined XML schema files). If the application's "payload" data file format already includes all necessary metadata, then the XFDU Package reduces to just the payload data file itself and a simple Manifest file. For Non-Val RM data, the payload data file is the TDM file. Among the things that have not yet been specified are whether or not there need to be any accompanying (or referenced) metadata files, and if so, the contents and/or locations of those metadata files. Can a provider pack more than one TDM into an XFDU Package, or should a new XFDU Package be generated for each TDM file (i.e., are there any timeliness requirements for this less-than-real-time service)?

TGFT is a push-only "protocol",  so once a NonVal TDM file is generated, the resulting TGFT XFDU Package must be created and pushed through the TGFT Sender to the TGFT Receiver associated with the target application. So the "service request"(so to speak) to the TGFT Sender entity has the arguments TGFT XFDU Package and the URL of the "in-tray" of that target TGFT Receiver. On the receiving side, the target application somehow "knows" to poll the in-tray for files being sent to it. The bottom line implication of this is that the XFDU Package file name has to convey enough information to allow the target application to recognize it. So other missing pieces of the puzzle are the naming convention that translates the name of the TDM file to the more-general file-naming conventions already specified for TGFT, and the method (e.g., managed parameters?) by which the TGFT Sender knows the URL of the "in-tray" of the target TGFT Receiver.

TGFT uses HTTP over TLS, also known as HTTPS. By my (almost nonexistent) knowledge of HTTP, the "client" must open a TCP connection and then send an HTTP request, at which point the server uses the same TCP connection to return the corresponding HTTP response. Presumably the TGFT Sender (in the ESLT) is the "client" in this case. I don't know how this paradigm is intended to be used in a push-only scenario. Does the HTTP request contain the XFDU Package itself (the POST method), or something else (such as just the URL of the XFDU Package that the TGFT Receiver subsequently uses to pull the file)? What does the TGFT Receiver send back as the corresponding HTTP response? It's not at all obvious to me how this would work, but perhaps to someone who has more than my Wikipedia-level understanding of HTTP it is obvious. In any case, this, too, should be included in the missing specification.

Unless and until we have "standard" answers to these questions, I am reluctant to state (as the SCCS-ARD currently does) that automated transfer of nonvalidated RM data TDM files over TGFT is a currently-available capability.

I would be very happy if someone already has the answers to all these questions and all that would be required would be to document them somewhere (where to document them would be a follow-on question).

Comments?

Best regards,
John

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/css-csts/attachments/20210821/d56b4401/attachment-0001.htm>