[Css-csts] Issue with using the Buffered Data Delivery Procedure for Real-time Radiometric Data

[John Pietras]

Members of the CSTSWG ---
In developing the Real-time Radiometric Data CSTS (RRD-CSTS) White Book,
I have come across an issue regarding how to use the Buffered Data
Delivery Procedure (BDDP) to deliver partitions of a Telemetry Data
Message (TDM). In brief, the issue is that the requirement to always
deliver the Metadata portion of the TDM, even when the radiometric data
is being delivered in the "real-time" delivery mode, may require
extensive derivation of the BDDP. In this note I'd like to describe the
issue in more detail, identify some possibly approaches to resolving it,
and solicit comments from the CSTSWG membership.

Note - unfortunately, we have two different uses of "real-time" at work
here. In "real-time radiometric data", it means that the data is made
*available* in real-time via periodic sampling (in contrast with waiting
until the end of the tracking pass and sensing all of the data in one
large file). In "CSTS real-time delivery mode", it refers to the fact
that the service ensures that it is *delivered* with bounded latency.
I'll attempt to keep the two uses unambiguous in the following
discussion.

First, let me describe the situation in enough detail to expose where
the issue lies. The plan has been to provide real-time radiometric data
delivery by streaming partitions of a TDM over a CSTS. These partitions
contain various combinations of TDM Header, Metadata, and Data sections.
(A more-detailed description of the mapping of TDM sections to RRD-CSTS
partitions can be found in the technical note "Streaming CCSDS Tracking
Data Messages Over a Cross Support Transfer Service" in the file
"Streaming_TDM_over_CSTS-071004.doc", which has been posted to the CCSDS
Collaborative Work Environment (CWE) > Cross Support Services Area (CSS)
> Documents > CSS-CSTS > CWE Private > Future Services using Toolkit
folder at URL
http://cwe.ccsds.org/css/docs/Forms/AllItems.aspx?RootFolder=%2Fcss%2Fdo
cs%2FCSS%2DCSTS%2FCWE%20Private%2FFuture%20Services%20using%20Toolkit )

My original plan was to have the RDD-CSTS "application" create the
content of the TDM, organize the content into partitions as described in
the attached file, and use the BDDP to move those partitions across the
RDD-CSTS service instance via the TRANSFER-DATA operation. In
Heppenheim, we affirmed that both the CSTS real-time and complete
delivery modes were desirable. We also realized that in the real-time
delivery mode, the service would be required to ensure that no Metadata
is lost when the BDDP is forced to discard a transfer buffer (for file
size compactness, Metadata sections are included in TDMs only when
changes to the metadata occur). 

Currently, the BDDP does not "know" anything about the content of the
data parameter of the TRANSFER-DATA operation; in particular, is would
not distinguish between Metadata and Data TDM content. So how do we make
this work? 

In no particular order, here are some possible approaches that have
occurred to me:

1. Simply apply the toolkit principles of  procedure extensibility and
derive a RRD procedure from the BDDP that is "Metadata aware" and
capable of ensuring that the Metadata is always sent even when transfer
buffers are discarded. If all TDM partitions were sent via the
TRANSFER-DATA operation, this would require quite a derivation from the
standard BDDP, and it's conformance to the toolkit would be in name and
specification only. In other words, I don't think that one would be able
to use an executable module from a standard toolkit library - the RRD
BPPD would pretty much have to be written and compiled as a new
procedure.

2. A variation of 1, above - extend the NOTIFY operation of the BDDP in
two ways:
a. Use the NOTIFY (instead of the TRANSFER-DATA operation) to deliver
the Header and Metadata sections; and b. Add the accumulated Metadata
information accumulated since the last-sent transfer buffer to the 'data
discarded due to excessive backlog' NOTIFY invocation that is
automatically sent by the BDDP.
While this approach also alters the BDDP, it appears to do so in a more
controlled, less invasive manner than approach 1 above. (Of course, that
just might be my perception, and they might be equally difficult to
implement).

3. Put the responsibility for metering the TDM partitions and ensuring
that the Metadata partitions in the application process that creates the
partitions and then uses the CSTS toolkit procedure to deliver them.
This approach essentially replicates the buffering logic outside of the
procedure, and essentially negates the usefulness of the procedure. I
dislike this one.

4. Use instances of two different procedures to deliver the data: the
BDDP for the Data sections of the TDM, and the Unbuffered Data Delivery
Procedure to deliver the Header and Metadata sections. The partitions in
these separate streams would be time-tagged (and/or given a sequence
number that applies to both procedure instances) so that the user
application could reconstruct the original sequence of the partitions
and re-form the TDM. As it is my understanding that there are not
timing-preservation requirements between different procedures of the
same service instance, this approach would require that the user keep
track of the various partitions coming out of the two procedure
instances. 

There are likely to be other approaches and/or variations on the above
four that are worth considering, so please don't hesitate to identify
them.

Of the four approaches that I have identified, I think I like #2 best.
It conforms to the goal of being able to create services by deriving
from already-defined procedures, and I think it does it in a way that
can be added without essentially redefining the whole Buffered Data
Delivery procedure. The user side still has to reconstruct the TDM from
both TRANSFER-DATA and NOTIFY invocations, but at least those will
always be delivered to the user in the correct sequence (which is not
guaranteed by approach 4). I will proceed with approach #2, but I am of
course willing to change it if we identify a better one between now and
next month's meeting.

Best regards,
John


John Pietras
Global Science & Technology, Inc. (GST)
7855 Walker Drive
Suite 200
Greenbelt, Maryland, USA
20770-3239
Direct:   +1-240-542-1155
GST Main: +1-301-474-9696
Fax:      +1-301-474-5970