[SIS-CFDPV1] Multiple file delivery request in a single Metada PDU and compressed delivery....

[구철회]

1) Yes, compression for telemetry data will be interesting. I’ve checked it with one of telemetry file and the compression rate is about 50% using zip software.
2) Good points! I concur to all your remarks. Let me suggest one change in Metadata PDU structure here.
  If source/destination file name supports TLV in addition to LV, it may have some rooms for adaptions to multiple type of input and output file.
   For example,
        Type 10 hex for single file transmission, so same as LV in current CFDP specification
       Type 11 hex for multiple file transmission separated by CSV or null separated and to be delivered as a plain file delivery
        Type 12 hex for multiple file transmission separated by CSV or null separated and to be delivered as single archived file
       Type 13 hex for directory content transmission
       Type 14 hex for web content transmission, e.g., http://www.amazon.com/
        and so on.

Best,
Cheol

From: Felix.Flentge at esa.int <Felix.Flentge at esa.int>
Sent: Thursday, December 8, 2022 6:45 PM
To: Tomaso.deCola at dlr.de; 구철회 <chkoo at kari.re.kr>
Cc: sis-cfdpv1 at mailman.ccsds.org
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi,


  1.  Compression: yes, I think this is very mission specific (available resources, file store implementations, file contents), e.g. ESA missions are currently mainly writing files of space packets for CFDP downlink. They could apply compression to TM packets if they want to; instruments may already have optimised data formats; the MMU could apply some compression; …
Furthermore, it might in general be preferred to apply compression also for storage and not just the file transfer.
So, I would keep compression outside the CFDP protocol. One way to implement compression for transfer only may be to implement it as part of the Filestore Interface.
  2.  Multiple-Files-In-Single-Transactions: For simplicity, I would recommend to keep the standard as is (at most a single file per transaction). I think a change to allow multiple files per transactions would have quite an impact as the standard is written with the assumption of having a single file (or none). I don’t see a big efficiency improvement with such a change (we would save a few metadata pdu). It is of course perfectly valid to concatenate or tar/zip files before sending. Private message to users (e.g., specifying the list of files and length which are sent) or even private file segment metadata might be used to support such practices. However, I would think that in situation where we really worry about the overhead of metadata PDU, we might also think about applying CFDP in general.
If the concern is more about the need to have multiple proxy put file operations for downlinking multiple files, I think it is already possible to have multiple proxy operations in a single metadata.

Regards,
Felix

From: SIS-CFDPV1 <sis-cfdpv1-bounces at mailman.ccsds.org<mailto:sis-cfdpv1-bounces at mailman.ccsds.org>> On Behalf Of Tomaso de Cola via SIS-CFDPV1
Sent: 08 December 2022 09:53
To: chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: Re: [SIS-CFDPV1] Multiple file delivery request in a single Metada PDU and compressed delivery....

For the compression matter, what I’m wondering is whether this operation cannot be directly done at data archiving level prior to invoking the CFDP service, so that CFDP is solely responsible for the data transmission no matter how and what content is transported in the corresponding transaction. I think this is in any case mission-specific and the other people here in the mailinglist can say what they think in relation to their agency missions.

Regards,

Tomaso

From: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Sent: Donnerstag, 8. Dezember 2022 09:36
To: Cola, Tomaso de <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

No. You pointed out a valuable aspect of CFDP for actual use cases in space. If a file is requested in real-time manner, it must be treated as single CFDP transaction as current CFDP specification states.

Here, I would like to suggest another possible use case for delivering separated files but logically correlated, so these files can be archived by compression tools, tar or gzip, and be delivered to a receiving entity as single file. Thanks to compression, this scheme will enable efficient bandwidth in a link when sending entity has enough processing power for real-time compression for requested multiple files.

For instance, Filestore request has some Action codes, ‘0000’ for “Create File”. ‘0101’ for “Create Directory”.
Then, maybe “Copy directory” action/service won’t be necessary or useful for some conditions?

Best,
Cheol

From: Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de> <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Sent: Thursday, December 8, 2022 5:10 PM
To: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

About the specification of multiple file names this is certainly doable, although not presently supported (so that some modification of the blue book will be in any case necessary). My doubt is the overall effectiveness gain you achieve with this given the current CFDP protocol specification. If I understand correctly your idea, all files back to back will be accommodated in a single FDU, which will be turn into a number of CFDP PDUs and in turn transmitted over a convenient UT. Then I also expect that the FDU will be delivered only when you receive the EOF, i.e. the first file being transmitted will be available only when all the other files will be received. As such I’d see more effective to have instead separated FDUs for each file, i.e. dedicated transaction for each file transmission that is what CFDP does nowadays.

Am I missing something of your proposal?

Regards,

Tomaso

From: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Sent: Donnerstag, 8. Dezember 2022 01:20
To: Cola, Tomaso de <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi Tomaso.

Thanks for sharing your thoughts.

Specifying multiple file name wouldn’t be so complicated, For example, I guess it could be done if
Source file name (LV) --> source file name (TLV), where type is for CSV (comma-separated values).
So multiple file name can be given like “xx(type code for CSV)” “30” “aaa.txt,bbb.tlm,ccc.img,ddd.dat”. This can give some potential flexibility on operating CFDP for ground operators and spacecraft itself.

If the compression happens in outside of CFDP app layer, someone must extract the compressed CFDP packet itself to see how the CFDP header is configured.
Or, actually the compression can be done to File Data PDU than the target file itself.

Best,

Cheol

From: Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de> <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Sent: Wednesday, December 7, 2022 5:05 PM
To: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi Cheol,

My understanding of CFDP is that it’s transaction based, where a transaction is aimed at transferring a single file. As such, transmission of multiple files has to go over different transactions. Moreover, the metadata allow to specify a single file name (source and destination) so that the possibility of sending multiple files with the same put request does not seem to me possible.
As to the compression, in my opinion this is something that can happen outside the CFDP operations.

Regards,

Tomaso

From: SIS-CFDPV1 <sis-cfdpv1-bounces at mailman.ccsds.org<mailto:sis-cfdpv1-bounces at mailman.ccsds.org>> On Behalf Of ??? via SIS-CFDPV1
Sent: Mittwoch, 7. Dezember 2022 01:49
To: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: [SIS-CFDPV1] Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi, CFDP fellow.

Recently, I am questioning to myself about how I am able to get multiple files from remote space storage through CFDP.
In the current CFDPv1 specification, we have to send multiple put.request commands to a sending entity as many as needed.

Is it possible to think putting multiple file source in a metadata pdu and also possibly compressed by tar, zlib and so on to make it possible?
For example, we can compile a metadata pdu as follows when a sending CFDP entity got requests for the aaa.txt bbb.tlm, ccc.img, ddd.dat separately:
File size : file size of compressed file, i.e., size of measurement_data.gz
Source file name : aaa.txt bbb.tlm, ccc.img, ddd.dat
Destination file name : measurement_data.gz

In my opinion, voluntary compression service by a sending CFDP entity would be good in terms of bandwidth reduction, lower retransmission rate and so on.

Cheol


This message is intended only for the recipient(s) named above. It may contain proprietary information and/or protected content. Any unauthorised disclosure, use, retention or dissemination is prohibited. If you have received this e-mail in error, please notify the sender immediately. ESA applies appropriate organisational measures to protect personal data, in case of data privacy queries, please contact the ESA Data Protection Officer (dpo at esa.int<mailto:dpo at esa.int>).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/sis-cfdpv1/attachments/20221209/2d05e179/attachment-0001.htm>