[Css-csts] Issue with using the Buffered Data Delivery Procedure for Real-time Radiometric Data

Mon Feb 25 02:31:55 EST 2008

Dear John,
Thank you for sharing your concern.

I read the various proposals and please find below my thinking.

1. This first option implies a complete new procedure that would have the 
knowledge of what can be discarded and what cannot be discarded. I am not 
very enthusiastic with that option. So, I agree with you.

2. If I understand correctly, the header and meta-data would be 
transferred with the  NOTIFY operation. In case data would be discarded, 
the "data discarded" notification would be extended to carry the missing 
header and meta-data.
I see the following issue. How will the service know what data was 
discarded and what data need to be repeated in the "data discarded" 
notification?
The disadvantage of that option is that we (sort-of) misuse the 
notification for carrying data.
An alternative would be to transfer the header and meta-data using the 
TRANSFER-DATA operation and only use the "data discarded" notification to 
complete the missing data.

3. I don't like this approach either.

4. The use of two different instances is also quite tempting but I would 
propose a first BFFD instance for delivering the data and a second 
instance of the BFFD for delivering the header and meta-data. The second 
instance would be used with the complete mode ensuring that no data is 
discarded. As you state, each message is time tagged so that the user can 
reconstruct the correct sequence.
The advantage of that option is that data would come cleanly from two 
procedure instances. The disadvantage is that the user must re-sequence 
the data.

In my view only option 2 and 4 viable and should be studied. Could you 
please let me know your opinion regarding the option 2 alternative I 
proposed. Do you consider the option 4 not viable?

Another option (5) would to systematically add at the beginning of the 
transfer buffer the header and the meta-data so that even if a buffer is 
discarded the information is transferred with the next transfer buffer. I 
do not have the TDM format with me and I am not sure if that option would 
be acceptable. What do you think?

Best regards

Yves

"John Pietras" <john.pietras at gst.com> 
Sent by: css-csts-bounces at mailman.ccsds.org
15/02/2008 18:15

To
<css-csts at mailman.ccsds.org>
cc

Subject
[Css-csts] Issue with using the Buffered Data Delivery Procedure for 
Real-time Radiometric Data

Members of the CSTSWG ---
In developing the Real-time Radiometric Data CSTS (RRD-CSTS) White Book,
I have come across an issue regarding how to use the Buffered Data
Delivery Procedure (BDDP) to deliver partitions of a Telemetry Data
Message (TDM). In brief, the issue is that the requirement to always
deliver the Metadata portion of the TDM, even when the radiometric data
is being delivered in the "real-time" delivery mode, may require
extensive derivation of the BDDP. In this note I'd like to describe the
issue in more detail, identify some possibly approaches to resolving it,
and solicit comments from the CSTSWG membership.

Note - unfortunately, we have two different uses of "real-time" at work
here. In "real-time radiometric data", it means that the data is made
*available* in real-time via periodic sampling (in contrast with waiting
until the end of the tracking pass and sensing all of the data in one
large file). In "CSTS real-time delivery mode", it refers to the fact
that the service ensures that it is *delivered* with bounded latency.
I'll attempt to keep the two uses unambiguous in the following
discussion.

First, let me describe the situation in enough detail to expose where
the issue lies. The plan has been to provide real-time radiometric data
delivery by streaming partitions of a TDM over a CSTS. These partitions
contain various combinations of TDM Header, Metadata, and Data sections.
(A more-detailed description of the mapping of TDM sections to RRD-CSTS
partitions can be found in the technical note "Streaming CCSDS Tracking
Data Messages Over a Cross Support Transfer Service" in the file
"Streaming_TDM_over_CSTS-071004.doc", which has been posted to the CCSDS
Collaborative Work Environment (CWE) > Cross Support Services Area (CSS)
> Documents > CSS-CSTS > CWE Private > Future Services using Toolkit
folder at URL
http://cwe.ccsds.org/css/docs/Forms/AllItems.aspx?RootFolder=%2Fcss%2Fdo
cs%2FCSS%2DCSTS%2FCWE%20Private%2FFuture%20Services%20using%20Toolkit )

My original plan was to have the RDD-CSTS "application" create the
content of the TDM, organize the content into partitions as described in
the attached file, and use the BDDP to move those partitions across the
RDD-CSTS service instance via the TRANSFER-DATA operation. In
Heppenheim, we affirmed that both the CSTS real-time and complete
delivery modes were desirable. We also realized that in the real-time
delivery mode, the service would be required to ensure that no Metadata
is lost when the BDDP is forced to discard a transfer buffer (for file
size compactness, Metadata sections are included in TDMs only when
changes to the metadata occur). 

Currently, the BDDP does not "know" anything about the content of the
data parameter of the TRANSFER-DATA operation; in particular, is would
not distinguish between Metadata and Data TDM content. So how do we make
this work? 

In no particular order, here are some possible approaches that have
occurred to me:

1. Simply apply the toolkit principles of  procedure extensibility and
derive a RRD procedure from the BDDP that is "Metadata aware" and
capable of ensuring that the Metadata is always sent even when transfer
buffers are discarded. If all TDM partitions were sent via the
TRANSFER-DATA operation, this would require quite a derivation from the
standard BDDP, and it's conformance to the toolkit would be in name and
specification only. In other words, I don't think that one would be able
to use an executable module from a standard toolkit library - the RRD
BPPD would pretty much have to be written and compiled as a new
procedure.

2. A variation of 1, above - extend the NOTIFY operation of the BDDP in
two ways:
a. Use the NOTIFY (instead of the TRANSFER-DATA operation) to deliver
the Header and Metadata sections; and b. Add the accumulated Metadata
information accumulated since the last-sent transfer buffer to the 'data
discarded due to excessive backlog' NOTIFY invocation that is
automatically sent by the BDDP.
While this approach also alters the BDDP, it appears to do so in a more
controlled, less invasive manner than approach 1 above. (Of course, that
just might be my perception, and they might be equally difficult to
implement).

3. Put the responsibility for metering the TDM partitions and ensuring
that the Metadata partitions in the application process that creates the
partitions and then uses the CSTS toolkit procedure to deliver them.
This approach essentially replicates the buffering logic outside of the
procedure, and essentially negates the usefulness of the procedure. I
dislike this one.

4. Use instances of two different procedures to deliver the data: the
BDDP for the Data sections of the TDM, and the Unbuffered Data Delivery
Procedure to deliver the Header and Metadata sections. The partitions in
these separate streams would be time-tagged (and/or given a sequence
number that applies to both procedure instances) so that the user
application could reconstruct the original sequence of the partitions
and re-form the TDM. As it is my understanding that there are not
timing-preservation requirements between different procedures of the
same service instance, this approach would require that the user keep
track of the various partitions coming out of the two procedure
instances. 

There are likely to be other approaches and/or variations on the above
four that are worth considering, so please don't hesitate to identify
them.

Of the four approaches that I have identified, I think I like #2 best.
It conforms to the goal of being able to create services by deriving
from already-defined procedures, and I think it does it in a way that
can be added without essentially redefining the whole Buffered Data
Delivery procedure. The user side still has to reconstruct the TDM from
both TRANSFER-DATA and NOTIFY invocations, but at least those will
always be delivered to the user in the correct sequence (which is not
guaranteed by approach 4). I will proceed with approach #2, but I am of
course willing to change it if we identify a better one between now and
next month's meeting.

Best regards,
John

John Pietras
Global Science & Technology, Inc. (GST)
7855 Walker Drive
Suite 200
Greenbelt, Maryland, USA
20770-3239
Direct:   +1-240-542-1155
GST Main: +1-301-474-9696
Fax:      +1-301-474-5970

_______________________________________________
Css-csts mailing list
Css-csts at mailman.ccsds.org
http://mailman.ccsds.org/cgi-bin/mailman/listinfo/css-csts

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ccsds.org/pipermail/css-csts/attachments/20080225/ac446dd7/attachment.htm