[SIS-CFDPV1] RE: [EDS Std Prep] notes on the blue book draft.

Burleigh, Scott C (312B) scott.c.burleigh at jpl.nasa.gov
Fri Oct 17 16:16:12 UTC 2014


[Hi, all.  It occurs to me that this technical conversation needs to be opened up to the working group.  If it’s a question we can resolve prior to the meetings in London, excellent; otherwise I think it needs to be the first order of business when we meet on Monday morning.  For those who are seeing this thread for the first time, please scroll to the bottom to understand the context.]

A further thought on this: while I still think it would be cleaner to add a Flags octet at the start of the file data segment PDU, that would break backward compatibility with version 0 of CFDP: it would no longer be possible for a CFDP entity to emit compliant version-1 data that a version-1-unaware CFDP entity could consume.

A possibly less disruptive implementation:

·         The 25th bit of the PDU header reverts to being reserved.

·         The 29th bit of the PDU header is used to indicate “PDU control flag” (instead of “Segment Metadata flag”), a Boolean that is always ‘0’ and ignored for File Directive PDUs.

·         When the PDU Control flag is set to 1 in a file data segment PDU’s header, the first octet of the PDU’s contents is a Control Field comprising:

o   Checksum type (1 bit): 0 = modular checksum, 1 = CRC32.

o   Record continuation state (2 bits) as currently defined except that no segmentation control flag is needed.  If the record continuation state is ‘00’ then its interpretation depends on the record continuation state of the file data segment with highest offset, preceding the offset of this segment, whose record continuation state is non-zero.  If such a segment exists and its record continuation state is ‘01’ then this segment is the continuation of a record begun with that segment; otherwise, this segment is not part of any record.

o   Segment metadata length (5 bits).  The segment can contain a maximum of 31 octets of segment metadata (rather than 63 bytes).

This design would enable a CFDP entity to emit compliant version-1 data that a version-1-unaware CFDP entity could consume:

·         The PDU Control flag in each PDU header would be set to zero, so file data segment Control Fields would be omitted and the format of the file data segments would be version-0-compliant.

·         The Large File bit in each PDU header would be set to zero, so all size and offset fields would be 32 bits in length, which is version-0-compliant.

·         The first bit of the Metadata PDU contents would be zero, which would indicate “Record boundaries respected”, but this would be harmless at the receiving entity; I think nothing in the CFDP spec requires that the receiving entity act on the value in this field in any way.

·         The Closure Requested bit in the Metadata PDU contents would be zero – no problem as this bit is Reserved in version 0, so the receiving entity would not see or process it.

Of course it could be argued that the installed base for CFDP is still so small that we shouldn’t worry about this, in which case we can be a little more free in fiddling with these formats.  I’m open to that if everyone is comfortable with it.

What do you think?

Scott

From: Burleigh, Scott C (312G)
Sent: Friday, October 10, 2014 9:49 AM
To: 'Stuart Fowell'; 'Matt Jordan'; 'Chris Taylor'; 'Mr. Dai Stanton'; 'Dai Stanton'; 'Keith L. Scott'
Subject: RE: [EDS Std Prep] notes on the blue book draft.

Guys, I have been looking more closely at this and have got a suggestion that I think will work better.

The problem with simply relocating the checksum type from the EOF PDU to the Metadata PDU is that the Metadata PDU might in some cases arrive after the first file data PDU, due to loss/retransmission or to some sort of routing weirdness.  The correct checksum type really needs to be known whenever a file data PDU arrives; otherwise we could easily start computing the checksum using the wrong checksum type and never match up with the checksum in the EOF.

So I went looking for a spare bit in the CFDP PDU header and found none; we are now using all of them.

What I did notice, though, is that two of the PDU header bits are being used for data values that pertain only to file data segments, never to file directive PDUs: segmentation control and segment metadata flag.

I now think we should bite the bullet and just add a Flags octet at the start of the file data segment PDU.  This will enable us to release two bits back to the universal PDU header, for future use, and it will give us a place to note all of the file-data-segment-specific information:

·         Segmentation control

·         Segment metadata

·         Checksum type

·         5 spare bits for future use

This does, of course, add a byte of overhead to every file data segment.  When the segments are of maximum size this will increase transmission overhead by .0015%, which is pretty small but not zero.

The trade-off here, I think, is between adding a small amount of transmission overhead versus adding some complexity and additional processing time by requiring that the checksum be calculated or recalculated at the conclusion of file data segment reception in some cases.

If we weren’t introducing multiple checksum types I think we could have gotten away with just using those spare PDU header bits for the new features.  But now I think we’re adding enough new functionality that we’re faced with having to pay for it somehow.

Your thoughts?

Scott

From: Burleigh, Scott C (312G)
Sent: Monday, September 15, 2014 8:25 AM
To: Stuart Fowell; Matt Jordan; Chris Taylor; Mr. Dai Stanton; Dai Stanton; Keith L. Scott
Subject: RE: [EDS Std Prep] notes on the blue book draft.

Actually, at the moment there are 7 spare bits in the Metadata PDU.  I can’t think of any good reason not to relocate the checksum type from the EOF PDU to the Metadata PDU.

I think the decision to limit remote Suspend/Resume to Acknowledged mode was taken at the San Antonio meeting in late 2013.  I don’t recall the rationale, but I remember Juan had recommended this provision prior to discussions at that meeting.  I’ve got no opinion on this; I’m happy either to retain it, revise it, or remove it.

Scott

From: Stuart Fowell [mailto:Stuart.Fowell at scisys.co.uk]
Sent: Monday, September 15, 2014 7:58 AM
To: Matt Jordan; 'Scott.Burleigh at jpl.nasa.gov'; Chris Taylor; Mr. Dai Stanton; Dai Stanton; Keith L. Scott
Subject: Re: [EDS Std Prep] notes on the blue book draft.

Feedback from implementation/testing of the proposed CFDP updates, for discussion

Sent from my iPhone

On 15 Sep 2014, at 15:22, "Matt Jordan" <Matt.Jordan at scisys.co.uk<mailto:Matt.Jordan at scisys.co.uk>> wrote:





File Checksums
1.1.1        Checksum Procedures
The checksum shall be 32 bits in length.  It shall be either a CRC32 checksum or a modular checksum, depending on the applicable checksum type.  For checksum computation at the sending entity, the applicable checksum type shall be the checksum type specified in the Management Information Base remote entity configuration information pertaining to the receiving entity.  For checksum computation at the receiving entity, the applicable checksum type shall be the checksum type specified in the EOF PDU.





Having the checksum type carried in the EOF PDU is not efficient as it postpones the checksum calculation until after all the file data PDUs have arrived at the receiver. This has the effect of delaying the sending of ACK/FINISHED PDUs. The implication is that timeouts for inactivity and closure response must take account of that processing time.



Since files may be very large and since we can no longer generate the checksum as the transaction progresses this makes a potentially big task in terms of memory use and i/o at the receiving end. I have noted that on my PC it can take a very long time to generate a checksum for a large file. A bit of experimenting produced results of 5 to 15 seconds for a 5 gig file on my PC.



My implementation uses the java built in CRC32 functionality in the class java.util.zip.CRC32. This is efficiently implemented using calls to native code across JNI.



I try to mitigate this behaviour by generating the CRC of the default type (MODULAR) at the receiver as the file data arrives so if the requested type is MODULAR then the CRC is ready on arrival of EOF PDU but if the requested type is not MODULAR then the CRC must be recalculated from scratch.



It would be preferable to carry the checksum type in the metadata pdu thus we may know all there is to know about the transaction as early as possible; there are 6 spare bits in the Metadata PDU.





Remote Suspend Transmission Mode



The highlighted text below implies that you may not remotely suspend a transaction if you do not have a return path. Is this correct? It seems that if you have a reliable UT layer then this is unnecessary.
6.5.3 Initiating a REMOTE Suspend
6.5.3.1 General
When required to initiate a remote suspend operation, the CFDP user shall use the CFDP Put.request primitive to request delivery of a file delivery unit whose metadata contains the Reserved CFDP Message defined in 6.5.3.2.  The transaction mode of the Put.request primitive shall be Acknowledged.
Initiating a REMOTE resume
General
When required to initiate a remote resume operation, the CFDP user shall use the CFDP Put.request primitive to request delivery of a file delivery unit whose metadata contains the Reserved CFDP Message defined in 6.6.3.2.  The transaction mode of the Put.request primitive shall be Acknowledged.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/sis-cfdpv1/attachments/20141017/c29ce2c8/attachment.html>


More information about the SIS-CFDPV1 mailing list