[SIS-CFDPV1] Multiple file delivery request in a single Metada PDU and compressed delivery....

Fri Dec 9 00:10:54 UTC 2022

Good points!
As far as I can see, there are two ways to do it.
1) Compression by payload : A science payload produces multiple files or single file of science measurement data. Possibly the science measurement data can be compressed at creation time. Anyway for multiple files delivery, payload management S/W has to pact these files to single tar.gz file. And make a put.request for CFDP S/W and wait until all transaction is finished before deleting the temporary tar.gz for securing storage space for next mission.
2) Compression by CFDP S/W : A science payload produces multiple files or single file of science measurement data (same as above). But payload management S/W doesn’t have to create a tar.gz file. Payload management S/W just requests single/multiple file delivery to CFDP S/W. Then CFDP S/W creates a temporary tar.gz, delivers it to receiving entity and deletes it after all transaction is finished.

Of course, payload engineer will prefer method 2 than method 1 while CFDP developers hate it. ;-)
Good talk.

Best,
Cheol

From: Tomaso.deCola at dlr.de <Tomaso.deCola at dlr.de>
Sent: Thursday, December 8, 2022 5:53 PM
To: 구철회 <chkoo at kari.re.kr>
Cc: sis-cfdpv1 at mailman.ccsds.org
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

For the compression matter, what I’m wondering is whether this operation cannot be directly done at data archiving level prior to invoking the CFDP service, so that CFDP is solely responsible for the data transmission no matter how and what content is transported in the corresponding transaction. I think this is in any case mission-specific and the other people here in the mailinglist can say what they think in relation to their agency missions.

Regards,

Tomaso

From: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Sent: Donnerstag, 8. Dezember 2022 09:36
To: Cola, Tomaso de <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

No. You pointed out a valuable aspect of CFDP for actual use cases in space. If a file is requested in real-time manner, it must be treated as single CFDP transaction as current CFDP specification states.

Here, I would like to suggest another possible use case for delivering separated files but logically correlated, so these files can be archived by compression tools, tar or gzip, and be delivered to a receiving entity as single file. Thanks to compression, this scheme will enable efficient bandwidth in a link when sending entity has enough processing power for real-time compression for requested multiple files.

For instance, Filestore request has some Action codes, ‘0000’ for “Create File”. ‘0101’ for “Create Directory”.
Then, maybe “Copy directory” action/service won’t be necessary or useful for some conditions?

Best,
Cheol

From: Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de> <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Sent: Thursday, December 8, 2022 5:10 PM
To: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

About the specification of multiple file names this is certainly doable, although not presently supported (so that some modification of the blue book will be in any case necessary). My doubt is the overall effectiveness gain you achieve with this given the current CFDP protocol specification. If I understand correctly your idea, all files back to back will be accommodated in a single FDU, which will be turn into a number of CFDP PDUs and in turn transmitted over a convenient UT. Then I also expect that the FDU will be delivered only when you receive the EOF, i.e. the first file being transmitted will be available only when all the other files will be received. As such I’d see more effective to have instead separated FDUs for each file, i.e. dedicated transaction for each file transmission that is what CFDP does nowadays.

Am I missing something of your proposal?

Regards,

Tomaso

From: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Sent: Donnerstag, 8. Dezember 2022 01:20
To: Cola, Tomaso de <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi Tomaso.

Thanks for sharing your thoughts.

Specifying multiple file name wouldn’t be so complicated, For example, I guess it could be done if
Source file name (LV) --> source file name (TLV), where type is for CSV (comma-separated values).
So multiple file name can be given like “xx(type code for CSV)” “30” “aaa.txt,bbb.tlm,ccc.img,ddd.dat”. This can give some potential flexibility on operating CFDP for ground operators and spacecraft itself.

If the compression happens in outside of CFDP app layer, someone must extract the compressed CFDP packet itself to see how the CFDP header is configured.
Or, actually the compression can be done to File Data PDU than the target file itself.

Best,

Cheol

From: Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de> <Tomaso.deCola at dlr.de<mailto:Tomaso.deCola at dlr.de>>
Sent: Wednesday, December 7, 2022 5:05 PM
To: 구철회 <chkoo at kari.re.kr<mailto:chkoo at kari.re.kr>>
Cc: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: RE: Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi Cheol,

My understanding of CFDP is that it’s transaction based, where a transaction is aimed at transferring a single file. As such, transmission of multiple files has to go over different transactions. Moreover, the metadata allow to specify a single file name (source and destination) so that the possibility of sending multiple files with the same put request does not seem to me possible.
As to the compression, in my opinion this is something that can happen outside the CFDP operations.

Regards,

Tomaso

From: SIS-CFDPV1 <sis-cfdpv1-bounces at mailman.ccsds.org<mailto:sis-cfdpv1-bounces at mailman.ccsds.org>> On Behalf Of ??? via SIS-CFDPV1
Sent: Mittwoch, 7. Dezember 2022 01:49
To: sis-cfdpv1 at mailman.ccsds.org<mailto:sis-cfdpv1 at mailman.ccsds.org>
Subject: [SIS-CFDPV1] Multiple file delivery request in a single Metada PDU and compressed delivery....

Hi, CFDP fellow.

Recently, I am questioning to myself about how I am able to get multiple files from remote space storage through CFDP.
In the current CFDPv1 specification, we have to send multiple put.request commands to a sending entity as many as needed.

Is it possible to think putting multiple file source in a metadata pdu and also possibly compressed by tar, zlib and so on to make it possible?
For example, we can compile a metadata pdu as follows when a sending CFDP entity got requests for the aaa.txt bbb.tlm, ccc.img, ddd.dat separately:
File size : file size of compressed file, i.e., size of measurement_data.gz
Source file name : aaa.txt bbb.tlm, ccc.img, ddd.dat
Destination file name : measurement_data.gz

In my opinion, voluntary compression service by a sending CFDP entity would be good in terms of bandwidth reduction, lower retransmission rate and so on.

Cheol

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/sis-cfdpv1/attachments/20221209/5ddf9b00/attachment-0001.htm>