[Css-csts] RE: Buffered Data Processing Procedure

Tue Feb 5 17:12:34 EST 2013

Martin,
Here are my replies on most of the items, in red.

Best regards,
John

From: Martin Götzelmann [mailto:martin.goetzelmann at vega.de]
Sent: Tuesday, January 29, 2013 4:00 PM
To: John Pietras
Cc: CCSDS_CSTSWG (css-csts at mailman.ccsds.org)
Subject: RE: Buffered Data Processing Procedure

Dear John,

Many thanks for your comments - be assured that they are really appreciated. My responses below in black color and italic typeface.

1.       The name of the parameter in the PROCESS-DATA invocation is "data", but the DP and BDP specifications are written (almost) universally in terms of the "data unit" of the PROCESS-DATA invocation. The change in terminology between operation and procedure might be interpreted as a refinement of the content of the data parameter for the purposes of the procedures, but in fact I don't think that there really is any difference in the content or format of either (i.e., the 'data unit' is still an abstract entity requiring further specification. Is there is reason that "data unit" is used in the procedures and not just "data"?

[MG] The term "data unit" has indeed not intended to refer to a refinement of "data"; I have adopted this terminology from the original specs without much thinking. For me the sentences sound better when using "data unit" but that might just be due to my limited understanding of the English language. So far I have not considered this a problem - as an example, the parameter in the CLTU TRANSFER-DATA operation carrying the CLTU is called "data" but the book talks about CLTUs everywhere and not about "data".

[JP] I agree that "data unit" sounds better, and it had just struck me as interesting that the term that we found most appropriate to discuss it is a bit different from the name of the actual parameter, and in this case I wondered if there was some deliberate reason for that. (Had I been involved in the writing of the original FCLTU book, I might have suggested that the CLTU-TRANSFER-DATA parameter be called "cltu" instead of "data"). But it's really not important and I won't discuss it any more.

2.       There are five parameters identified throughout the two procedures that are used to constrain or direct the behavior of the procedures: (1) the size of the input queue (identified as being defined by the service using the procedure), (2) the size of the input queue (identified as being defined by the service that uses the procedure), (3) the BDP transfer mode (how it is set is not identified), (4) maximum transfer buffer size (identified as a managed parameter), and (5) processing-latency-limit (identified as a "global" managed parameter, a managed parameter, and as being specified by the service that uses the procedure - more on this in a moment). The "Parameter Values to be Specified Outside of the Framework" annex of the CSTS SFW is a repository for such parameters - when the DP and BDP procedures are eventually integrated into the CSTS SFW, these parameters should be included in the cited Annex. Regarding the processing-latency-limit parameter, you (Martin) included a comment asking whether being labeled as a "managed parameter" implied being set by Service Management and if so, how that affects the subsequent statement that this parameter is set by the service that uses the procedure. My opinion is that calling something a managed parameter *does* imply Service Management. If we need a more-generic term, we could call such a parameter a "configuration parameter" or "configurable parameter", and then state the method by which it is set. The most general method is to set by means specified by the service that uses the procedures, because such a service can always specify that for that service the parameter is managed. Only if we are sure that a configuration/configurable parameter should always be set by management for all direct and derived applications of a procedure should the specification of that procedure declare a parameter to be managed. Note that deferring how a configurable parameter is set to the derived service also implies that the service can fix such a parameter to a single value - e.g., a service could be defined that uses only the Complete transfer mode (Note, too, that the Transfer Mode should be identified in the BDP specification as a configuration/configurable parameter and a statement made about how it is set (e.g., deferred to the service that uses the procedure)).

[MG] This should be discussed in combination with item 14.

Do we have general rules how configuration parameters are called and described?  If "managed parameter" implies that the parameter is handled by Service Management then I will change the name  - I can use configuration parameter, but in the SFW this term is only used in section 2.3 in the context of configuration management. My intention has been that all of these are defined by the service using the procedure including the options that the service may specify a fixed value in the specification or delegate the definition to service management. I do think I have specified that everywhere (for  BDP transfer mode, see 1.1.4.2.3.2 "The transfer mode to be applied shall be defined by the service using the procedure" and for the Transfer Buffer size see 1.1.4.2.1.2 ).

[JP] Perhaps it's not necessary to give a name (e.g., "configuration parameter") to these parameters, but merely state that they shall be defined by the service (or derived procedure) while reserving the term "managed parameter" for those parameters (if any) that we know must be managed.

The term "general managed parameter" meant to say that there is a single value that applies to all PD invocations and not specific values for individual invocations (as for the CLTU delay for instance). This can be clarified.

[JP] Since the word "global" is used only in this one case, it might be best to just not use it and avoid coining a phrase. It may not even be necessary to explicitly state for this one parameter that the single value applies to all PD invocations, because that seems to be the only way for such parameters to work, given that they are *not* included in the invocations themselves (which would be the only way to make them adjustable on a per-invocation basis.

I have briefly skimmed through the procedure specifications in the SFW and its Annex H. I have the feeling, we might need to have another review cycle to ensure consistency. A few questions and observations resulting from that exercise (and I am prepared to accept that it is too late to raise these questions!):

a)      Annex H includes configuration parameters but also not fully operation parameters that simply need to be specified by derived procedures. I feel that specifying format and semantics of a an undefined parameter and setting the value of a configuration parameter are very different and including these in one table is questionable.

b)      The Buffered Data Processing procedure says that configuration parameters to be applied for a service instance (e.g. delivery mode, buffer size) are defined by service management. Does this exclude using the procedure in a service that does not support the timely mode? What is the reason for not delegating the decision to the service using the procedure? I Assume the specification excludes that different delivery modes are used in the same service instance but on a quick check I did not find where that is said.

c)       In the Cyclic Report procedure the Parameter Lists are defined by service management. Does this exclude using the procedure in a service that just wants to report a fixed set of service parameters?

[JP] This is a good catch. We realized the equivalent possible situation for the use of the Notification procedure, which is why the creation of lists of events is phrased in terms of being defined by the service (as noted below). I agree that this should probably be changed for Cyclic Report. Note too that this same problem exists for the list-of-parameters parameter of the GET operation, although the Info Query procedure (which of course uses the GET operation) states that the specification of the contents of a parameter list is controlled by the service that uses the procedure.

This question of whether to simply defer to the service or explicitly cite Service Management has an interesting consideration concerning the Parameter Names that are requested and reported via the Cyclic Report and Information Query procedures (and any other procedure that uses the GET operation) and the Event Names that are requested and reported via the Notification procedure (and any other procedure that uses the NOTIFY procedure). The Parameter Names and Event Names include the Functional Resource Identifier, which in turn contains the Functional Resource Instance Number that is assigned by Service Management. That is, as currently defined, there is always an explicit Service Management involvement in the specification of these names. There is no way to create a service that uses one of these procedures or operations that does not have to rely on Service Management to complete the names of the parameters or events.  This is not necessarily a problem, but it is a change from the SLE approach. When an SLE service instance reports a change in the production status associated with that service instance, the identity of the SLE service instance is implicit in the association. A CSTS that reports its production status needs Service Management to tell it who it is (that is, its FR Identifier) before it can report its production status.

d)      In the Notification procedure the maximum number of notifiable events and the list of events is specified by the service using the procedure. The specification contains notes that the specification may be delegated e.g. to service management.

3.       [JP] no further comment

4.       DP procedure and Buffered Data Processing (BDP) procedure, multiple instances. The CSTS SFW has an informal typographical convention (that is, one that is not explicitly stated in 1.6.3.3) of naming formal entities (e.g., Release Timer, Transfer Buffer, Recording Buffer) with initial caps to distinguish them as terms with specific formal behaviors, characteristics, etc.. I suggest applying this convention to Input Queue and (for BDP) Latency Timer. [NOTE regarding CSTS SFW - should we formalize this convention in 1.6.3.3?]. Also, should the BDP specify the units of the Latency Timer (presumably, seconds)?

[MG] OK for capitalization - if this is a convention, then I think it should be spelled out because otherwise you cannot expect editors to apply it.

[JP] I agree that it should be documented. If the CSTSWG agrees, I would volunteer to write a draft convention for inclusion in 1.6.3.3.

As concerns the units for the latency timer see 1.1.4.2.2.1 - I have adopted the milliseconds specified in the previous version by Tim.

[JP] Oops, somehow I missed that.

5.       DP procedure, section 2.1.2.2 (Concept), end of third paragraph "This ['production interrupted' notification] report informs the user that the input queue may fill up and cause the service instance to be aborted if the Production Status remains in a non-operational state." There is no normative behavior specified regarding aborting the service if the queue fills while Production Status is non-operational.

[MG] I recognize my initial PEER ABORT got lost. What about using the same statement as in the note to 2.1.3.3.8.4 "The notification has the purpose to make the Service User aware that the processing of the data unit will be delayed until the Production Status becomes 'operational' and that the input queue might fill up if further PROCESS-DATA operations are invoked"?

[JP] That sounds fine to me.

6.       DP procedure, 2.1.3.2.5, NOTE, states "The Service Provider queues the PROCESS-DATA invocations so that the Service User may transfer the PROCESS-DATA invocations well in advance of the processing taking place." However, this rationale is negated by the Timely Transfer mode of the BDP, which has the processing-latency-limit to limit how far in advance the invocations can be transferred. I think there is a more universal reason for the input queue, which could be stated as "The input queue allows the timing of the processing of decouples the timing of the processing of the PROCESS-DATA invocations form the transfer of those invocations." For services that use a derived procedure that does have the equivalent of the processing-latency-limit, that could include allowing long delays between transfer and processing. But it also applies to the Timely mode of BDP in cases where the process pops the invocations from the queue asynchronously with respect to how they are transferred.

[MG] Agreed - I will change to "The input queue decouples the timing of the processing of the PROCESS-DATA invocations from the transfer of those invocations."

[JP] Thanks. That sounds fine.

7.       DP procedure, 2.1.3.3.1 states "The Service Provider shall remove PROCESS-DATA invocations from the input queue and process the data units included as soon as possible ...". Is the actual processing a part of the derived procedure itself, or is it part of a production process that the DP (or DP-derived) procedure hands the invocation to? The distinction may be important for how service specifications are written using DP procedures. If the processing is part of the derived procedure itself, then services using such derived procedures will have to have the, behavior and state machines for those derived procedures include the production processing. If, on the other hand, the processing is considered to be a separate "state machine", the impact on the DP-derived procedure itself can be minimized.  The existence of the input queue supports this second interpretation if that's the one that applies: the queue is basically a mailbox connecting the data transfer process to the "processing" process. For the Forward Frames service, for example, my plans have been to adopt this second interpretation (I took a similar approach in defining the enhanced functionality of the EFCLTU service).

Note that the current wording of the requirements under 2.1.3.3 is still valid in either case, because "Service Provider" can be interpreted in the broader sense to be not only the transfer service provider but the complete service provider (i.e., the Complex).  But if  interpretation is that it is the procedure that does the processing then 2.1.3.3 contains requirements on the procedure, whereas if it is a separate production process that does the processing then 2.1.3.3 contains common requirements on all production processing associated with the DP-derived procedure, and perhaps we might want to make that distinction.

[MG] My (possibly naive) imagination has always been that the processing is part of the production and as you say the input queue is a sort of mailbox that  connects data transfer with the production process. It is not clear to me what the implications are and I suggest discussing this further on the next teleconference.

[JP] I agree. Let's hope that we can carve out some time to discuss it at the telecon.

8.       DP procedure, 2.1.4.2.2.2 (a), "the processing of the data completed without aborting ..." I think that deleting "without aborting" would not change the meaning of this statement. "Abort" has a special (and different) meaning in CSTS and its use in this sentence could cause confusion.

[MG] Agreed. Just for information, the source is CLTU 3.7.2.7.2 " 'radiated'-radiation of the CLTU completed without aborting: the CLTU is guaranteed to have been radiated nominally;"

[JP] I'd like to suggest to Fabienne and Wolfgang that they consider deleting "without aborting" in the next update of the FCLTU book.

9.       [JP] Item resolved - no further comment.

10.   DP procedure, 2.1.5.1, Table 2-4 (State Table): The entries {'procedure to association abort 'xxx''} should be {procedure to association abort 'xxx'} - no single quotation marks around procedure to association abort 'xxx'.

[MG] You are probably right - I have just adopted this from the input I had. However, I cross-checked the SFW and there the corresponding entries are entered as 'procedure to association abort 'xxx'' but do not include the curly braces. What is correct? Can you point me to place where these conventions are defined in CSTS? I know where they are for SLE, but for CSTS?

[JP] The conventions are defined in 1.6.3.2.6, Procedure State Table Subsection. According to item (6) of that section, simple actions are identified by enclosure by single quotes (' ') whereas compound actions are enclosed by curly braces ({ }). While the simple actions that comprise a compound action are expressed using the single quote notation, the compound action name itself does not use the single quotes.

In this particular instance, there are both a simple action 'procedure to association abort 'xxx'' and a compound action {procedure to association abort 'xxx'} that comprises the simple actions 'clear the input queue' and 'procedure to association abort 'xxx''.

11.   Buffered Data Processing (BDP) procedure, 1.1.2.2 (Concepts): My apologies if this was discussed in CSTSWG meetings that I didn't attend, but I'm having a hard time conceiving of a use case for having a latency limit when the BDP is operating in Complete mode. Was one ever expressed, or was this possible combination included just in case someone might come up with a useful application?

[MG] I do not think this was discussed at least there is nothing in the minutes. I have no use case to offer and included this specification only "just in case". I have no problem restricting it to the timely mode if that is preferred by the WG.

[JP] On one hand, I don't object to leaving it in "just in case". On the other hand, I did spend a bit of time thinking about how this would work, and I'm a bit concerned that someone will think that services that use the Complete mode *should* establish a finite value for this parameter. Perhaps the addition of a NOTE along the lines of "Services that require BDP procedure instances operating in the Complete mode to be able to take as long as necessary to process every and all data units may need to fix the value of the processing-latency-limit parameter to zero to ensure that no data units are discarded."

12.   Buffered Data Processing (BDP) procedure, 1.1.2.2 (Concepts):  It might be useful to have a diagram or two to illustrate the Transfer Buffer and Input Queue and the relationship(s) between them.

[MG] Well, I have never been too comfortable with the diagrams we have in the book for the Buffered Data Delivery  procedure but would of course include any diagram that may be "donated" by somebody else.

[JP] I think it will depend in part on how we address the question of whether the processing is considered part of the production or part of the provision. Once that's settled I might try to generate it/them for a (near) future draft.

13.   Buffered Data Processing (BDP) procedure, 1.1.2.2 (Concepts), last sentence of second paragraph: "The maximum size of the Transfer Buffer expressed as the number of PROCESS-DATA invocations that may be stored in the buffer is defined as a managed parameter." Suggest a slight rewriting for clarity, as follows "The maximum size of the Transfer Buffer is expressed as the number of PROCESS-DATA invocations that may be stored in the buffer. It is defined as a managed parameter."

[MG] Agreed, but in view of your pervious comment (2) we should probably say. "The maximum size of the Transfer Buffer is expressed as the number of PROCESS-DATA invocations that may be stored in the buffer. It is a configuration parameter that is defined by the service using the procedure."

[JP]  Yes.

14.   BDP procedure, 1.1.2.2 (Concepts), last sentence of second paragraph, states that the maximum transfer buffer size is expressed in terms of the number of P-D invocations. There is a relationship between the transfer buffer and the input queue, namely the contents of the transfer buffer aren't put on the input queue until the full buffer contents can be placed there. However,  the "units" of the input queue itself are not defined. Is the intention that the input queue also be specified in terms of P-D invocations? If so this could result in less data being queued than the input queue can physically accommodate, but that might be a secondary consideration if the actual number on the queue is expected to more important than the data volume. Note that if the input queue definition is going to be refined by the BDP procedure, then the statements in 2.1.2.2 (Concepts) and 2.1.3.2.6 of the DP procedure regarding the size of the input queue being defined by the service be modified to allow modification by a procedure that is derived from this procedure or a service that uses this procedure.

[MG] In my understanding these are two points:

a)       I suggest we specify that the input queue size is defined in terms of the maximum number of P-D invocations as I do believe the number of units is more important than memory space. We could also say "maximum sized" P-D invocations, as this is done in CLTU, but then it may be argued  that we do not specify a maximum size for the data in a P-D invocation. Obviously, the Transfer Buffer size must be smaller than the Input Queue size but I do not think we need to specify that.

b)      I think we need to agree how we want to deal with defining of parameters in the context of deriving procedures and using procedures in services. In my understanding a deriving procedure can redefine the approach to setting of parameters whatever the parent procedure has specified. For instance if the parent procedure says "defined by the service" a derived procedure could either fix the value or specify that it must go to service management. I think even if the parent procedure has assigned the parameter to service management a derived procedure could state that for the specific case the procedure handles it must be 4711. Do we therefore always have to say that a parameter shall be defined by a derived procedure or by a service using the procedure? Once we have agreed on a general statement we probably have to go through all FW procedures to ensure that they are consistent (see also item 2).

[JP] Hmmm - I had been thinking that if a procedure declares a parameter to be managed it binds it to being managed, but I see your point. Maybe we should just use the "managed parameter" designation vary sparingly in the SFW (perhaps not at all). I still think that it is good to identify that a parameter is one that needs the service or derived procedure to define how it is set, if for no other reason than it flags those parameters as ones that have to addressed by the service/derived procedure.  If we don't want to use the long phrase ("[parameter] shall be defined by a derived procedure or by a service using the procedure") than we might come up with a designation to flag such parameters. I had suggested "configuration parameter" or "configurable parameter" but I'm not partial to either of those.

15.   BDP procedure, 1.1.2.2 (Concepts), beginning of sixth paragraph, states "In Complete Transfer mode, the Service Provider stops reading data from the underlying data communication service when the available space on the input queue drops below the maximum size of a Transfer Buffer and resumes reading data when there is sufficient room on the input queue to store all PROCESS-DATA invocations that might be included in a maximum sized Transfer Buffer." Unless I am mistaken, operating a CSTS over ISP would have all procedures operating over the same TCP connection. If that is true, then if a service were to allow multiple concurrent instances of the BDP procedure, any one of them running in Complete mode would suspend transfer for all of them (even ones running in Timely mode, which would be absolutely counter to the purpose of Timely mode). I can't think of any reason why someone would want to create a service that would use multiple concurrent BDP procedure instances, but that doesn't mean that someone might not try it sometime in the future. The BDP procedure specification should (a) outright prohibit the use of multiple BDP instances where at least one of the instances is operating in the Complete mode or (b) include a note of some sort  identifying the consequences of defining a service that allows multiple instances of BDP to be instantiated if at least one of the instances can be in Complete mode.

[MG] You are of course right - you cannot use more than one BDPP in a service instance at least if one association is mapped to one TCP connection as this is done by ISP1. I do not think a procedure specification can prohibit use of multiple instances but it can point out what the consequences would be. If considered useful, I could  add another NOTE to 1.1.4.2.3.3 to say that.

[JP] Yes, I think a note pointing out the consequences is the right approach.

16.    BDP procedure, 1.1.2.2 (Concepts), last sentence of sixth paragraph, states "In effect, data will always be transferred completely accepting an additional delay above the processing latency controlled by the processing-latency-limit parameter." I do not understand what the "accepting additional delay" clause is attempting to say. This same phrasing is used in 1.1.4.2.3.1 NOTE 1.

[MG] What I wanted to say is that the purpose of the complete mode is to ensure that all data units are processed independent of the latency that might be implied. For the latency one has to take into account the time that the data are waiting on the input queue plus the transfer time plus the time the data might be waiting in the network. As you have noticed the specification allows constraining the time on the queue also for the complete mode (which we might want to change) but the additional delay caused by data waiting "in the communications network" cannot be avoided. I would be grateful for words that would express this more clearly.

[JP] I've started several attempts at trying to write an alternative description, but in each case have ended up wondering if what I'm writing addresses the point that you are trying to make or is even correct. Rather than delaying this response any longer on this point I'm going to skip over it for now and perhaps we can discuss it in further correspondence.

17.   [JP] no further comment.

18.   [NOTE - this comment is really a comment on the CSTS SFW itself, but is was raised by the following statements in the BDP specification] BDP procedure, 1.1.4.2.3.3, NOTE 2 states "In this situation [that is, when the Service Provider has stopped reading the data from the underlying comm service while in Complete mode] the Service User will not be able to terminate the procedure nominally by invocation of the STOP operation until all transmitted data have been read and queued by the Service Provider. The only way to terminate the procedure earlier is by invocation of PEER-ABORT." Terminating the procedure while in this suspended-reading condition via the PEER-ABORT is possible when the underlying comm service includes ISP-1 because ISP-1 uses TCP, which provides the urgent data feature. However, the capability to have a PEER-ABORT bypass the flow control of the underlying comm service is never explicitly called out in either the specification of the PEER-ABORT operation or the characteristics of the underlying comm service (1.3.1) of the CSTS SFW. There *are* statements in the CSTS SFW to the effect that ISP-1 is the "default" underlying protocol, but it's still not normative/mandatory in the text sections. The only place in the CSTS SFW that ISP-1 is essentially normative for CSTSes is in the ASN.1 specification of the PEER-ABORT diagnostics. The CSTS SFW should take one of two positions: (a) declare ISP-1 to be the *required* underlying protocol of all CSTSes, in which case the phrases such as "default" underlying service, "assumed to be used", and "applicable to" should be replaced by more direct statement (i.e., ISP-1 is the underling protocol'); or (b) still allow ISP-1 to be one of possibly several underlying protocols, in which case the characteristics of the underlying comm service in 1.3.1 need to be expanded to allow for the bypassing of flow control mechanisms.

[MG] In my understanding the PEER-ABORT specification requires that a diagnostic be delivered to the peer application why the association has been aborted, not that the invocation bypass the flow control of the underlying communication service. For instance, the old OSI transport service provided a feature by which user data could be passed as part of the procedure for disruptive teardown of an association. ISP-1 implements a work-around based on the TCP urgent byte because TCP does not support user data to be combined with RESET. Other approaches could be imagined - for instance one could just RESET the TCP connection and try to send a UDP datagram with the diagnostics or one could always use two TCP connections where one is only used to support PEER-ABORT. One can argue that the PEER-ABORT specification is too demanding and drop the diagnostics, but I do believe the SFW includes all specifications needed to build an alternative communication service if so desired.

[JP] I'll certainly defer to you regarding whether there might be other ways to provide the PEER-ABORT besides the TCP urgent byte.

I'm still not sure that there are enough *normative* statements to guarantee that an implementation will (a) move the PROCESS-DATA invocations/Transfer Buffers across a flow-controlled channel and (b) ensure that the PEER-ABORT will not get caught in the flow-controlled channel. There are informative statements about back-pressure, but where is it stated normatively?  If you tell me that you believe that it is all nailed down I will take your word for it.

19.   [JP] no further comment.

20.   BDP procedure, 1.6.1.1, table 1-2 (State Table): Some of the actions are italicized, and it appears that these italicized sections are those that are copied from the parent DP procedure. If this is the case a note explaining that would be helpful.

[MG] I have adopted this approach from the cyclic report procedure in the SFW and it has also been applied to the Sequence Controlled DPP. It has been introduced by Yves when we detected that we were re-specifying the state tables for derived procedures. Maybe the explanation should go to the text you have proposed on "how to read the specifications".

[JP] That sounds like a good resolution.

[JP] I have to wrap up for tonight and I want to get the above comments to you now. I'll respond to the following two topics and respond to them tomorrow morning.

I would also be interested in your opinion regarding the  presentation of derived procedures in general as reflected by the questions I have inserted as comments into the BDPP.

Finally, as you have not come back to the points in the Simple DPP for which I had included comments in version 10, can I assume that these are OK for you?

Kind Regards, Martin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ccsds.org/pipermail/css-csts/attachments/20130205/cb89f9a9/attachment-0001.html