<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Malgun Gothic";
panose-1:2 11 5 3 2 0 0 2 0 4;}
@font-face
{font-family:"\@Malgun Gothic";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>Right, that’s why I said “rare”. That is, from what I’ve been hearing about streaming I’m guessing it’s relatively rare.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Scott<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Keith Scott <keithlscott@gmail.com> <br><b>Sent:</b> Friday, February 16, 2024 9:08 AM<br><b>To:</b> sburleig.sb@gmail.com<br><b>Cc:</b> <span style='font-family:"Malgun Gothic",sans-serif'>구철회</span> <chkoo@kari.re.kr>; sis-dtn@mailman.ccsds.org<br><b>Subject:</b> Re: [Sis-dtn] ION and in-order delivery<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>But IP will deliver things out of order, just for different reasons. Generally it's due to routing or link bonding or something like that.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> --keith<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Fri, Feb 16, 2024 at 3:25 PM <<a href="mailto:sburleig.sb@gmail.com">sburleig.sb@gmail.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>My sense is that this issue would never have come up if the implementation of bplib had not optimized for storage management by storing bundles in non-contiguous locations (as space became available) but clocking them out in ascending location order rather than in reception/storage time order.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>This is behavior that you’ll never see in IP because IP never stores packets at all, it simply forwards them immediately upon reception. In this sense IP itself is non-delay-tolerant, i.e., it doesn’t compensate for unplanned lapses in connectivity by retaining data for transmission when connection is restored. The mis-ordering of data will be routine in the bplib implementation of BP but (suitably) rare in any implementation of IP.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>We call BP a store-and-forward or store-carry-forward protocol, but there actually isn’t any language that I can find in RFC 9171 that requires that bundles be stored at all; we just assume that we all know it has to happen. That’s an assumption that a developer picking up the specification might not share, so some sort of note pointing this out might be helpful.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Scott<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:currentcolor currentcolor'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b>From:</b> SIS-DTN <<a href="mailto:sis-dtn-bounces@mailman.ccsds.org" target="_blank">sis-dtn-bounces@mailman.ccsds.org</a>> <b>On Behalf Of </b>Keith Scott via SIS-DTN<br><b>Sent:</b> Thursday, February 15, 2024 11:11 PM<br><b>To:</b> <span style='font-family:"Malgun Gothic",sans-serif'>구철회</span> <<a href="mailto:chkoo@kari.re.kr" target="_blank">chkoo@kari.re.kr</a>><br><b>Cc:</b> <a href="mailto:sis-dtn@mailman.ccsds.org" target="_blank">sis-dtn@mailman.ccsds.org</a><br><b>Subject:</b> Re: [Sis-dtn] ION and in-order delivery<o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Carlo: yeah, it's a very simple scenario; I took an existing test and plotted bundle sequence number against receive order.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Cheol + Carlo: yes, I believe the misordering behavior is entirely allowed by the sentiment and letter of and specification. The way I understand BP, the service provided by BP is similar to that provided by IP:<o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> - Delivery of (bundles / datagrams) to the destination (name / address)<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>In particular, in both BP and IP, (bundles / datagrams) may be lost, duplicated, delayed, and/or misordered during delivery.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Context<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>=======<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Sorry I didn't provide context for the first message. We were discussing in the SIS-DTN telecon implementations that might want to impose a requirement along the lines of:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> "All other things being equal, a bundle node should forward bundles in FIFO order."<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>That's not a requirement in any of the BP specs, but in general it's probably beneficial to applications if things don't arrive too much out of order if they don't have to. Really this stems from folks who want to be able to:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> - In bpv6, set timers like custodial retransmission timers or timers associated with DTPC, etc.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> - Do something intelligent with application-layer timers related to lost chunks of a stream<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I think the above guidance to implementers is probably good, though there's (IMHO) a world of complexity hiding under "All other things being equal", especially when we consider QoS and the group's desire to be able to dynamically modify QoS treatments of bundles via policy.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> --keith<o:p></o:p></p></div></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Fri, Feb 16, 2024 at 1:40 AM <span style='font-family:"Malgun Gothic",sans-serif'>구철회</span> <<a href="mailto:chkoo@kari.re.kr" target="_blank">chkoo@kari.re.kr</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)'><div><div><div><p><span style='font-size:10.0pt'>Hi, Keith.<br><br>I think that's the normal operational behavior in LTP and TCPCL even in the absence of packet loss during connection because there are discontinuity events between A-B and B-C.<br><br>When a discontinuity event occurs during a transaction of a LTP session, the LTP session could possibly not be ended successfully and may pause to await the Report segment from the receiving entity. Upon reconnecting, the affected LTP session must resume interrupted session processing (e.g., RS-DS retransmit-RS-RAS), while other LTP sessions can continue smoothly.<br><br>Considering that an LTP session typically holds a bundle, in this scenario, out-of-order bundle delivery can occur even without packet loss. *NOTE* Discontinuity effectively simulates packet loss. TCPCL just acts like LTPCL. I think you could check it in a wireshark captured packet by seeing a Report segment.<br><br>In your scenario, discontinuity events occurred 8 times (4 times for A-B, 4 times for B-C). I can observe 8 times of bundle arrival fluctuation events in the figure.<br><br>Maybe there could be other issues originating this phenomenon, but above is my thought. HTH.<br><br>Best,<br>Cheol<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 15 Feb 2024 20:59:24 +0100<br>From: Keith Scott <<a href="mailto:keithlscott@gmail.com" target="_blank">keithlscott@gmail.com</a>><br>To: <a href="mailto:sis-dtn@mailman.ccsds.org" target="_blank">sis-dtn@mailman.ccsds.org</a><br>Subject: [Sis-dtn] ION and in-order delivery<br>Message-ID:<br> <CAHdkBBmXz6sQrY50p6Ob3ThO005132Ph9KZOiSgW+=<a href="mailto:C-8J5Y0w@mail.gmail.com" target="_blank">C-8J5Y0w@mail.gmail.com</a>><br>Content-Type: text/plain; charset="utf-8"<br><br>I ran a quick test with a 3-node ION network A-B-C where for the first minute A is connected to B (via LTP) and B is connected to C (via TCP).<br>After that the topology repeats every two minutes; during the first minute A is connected to B and during the second B is connected to C. At t=540s full connectivity (A-B and B-C) is restored for one minute. There is no artificially induced loss in this scenario. The max aggregation size for LTP was 100kbytes and the max aggregation time was 1s, e.g.:<br>* a span 2 10 10 64000 100000 1 'udplso <a href="http://10.44.3.2:1113" target="_blank">10.44.3.2:1113</a><br><<a href="https://protect2.fireeye.com/v1/url?k=ce37f0ae-91ac9aa6-ce328120-ac1f6bdccbcc-797d3695053eff4c&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2F10.44.3.2%3A1113%2F" target="_blank">https://protect2.fireeye.com/v1/url?k=ce37f0ae-91ac9aa6-ce328120-ac1f6bdccbcc-797d3695053eff4c&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2F10.44.3.2%3A1113%2F</a>> 1000000'*<br><br><br>[image: image.png]<br><br>I ran bping (will do a unidirectional test later). The chart below shows the received bping sequence number as a function of the order in which bpings were received.<br><br>[image: image.png]<br><br>So yeah, it looks like ION will occasionally misorder bundles. I think that's not ideal, but I strongly believe that it is compliant with both the spec and the intent of the service (bundle delivery) BP purports to provide. (And most bundles were in order, which is a feat considering that between 60 and 540s the network is never end-to-end connected).<br><br> --keith<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <<a href="https://protect2.fireeye.com/v1/url?k=f133e8de-aea882d6-f1369950-ac1f6bdccbcc-aa39b8d587d0d7dd&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2Fmailman.ccsds.org%2Fpipermail%2Fsis-dtn%2Fattachments%2F20240215%2F70f735de%2Fattachment.htm" target="_blank">https://protect2.fireeye.com/v1/url?k=f133e8de-aea882d6-f1369950-ac1f6bdccbcc-aa39b8d587d0d7dd&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2Fmailman.ccsds.org%2Fpipermail%2Fsis-dtn%2Fattachments%2F20240215%2F70f735de%2Fattachment.htm</a>><br>-------------- next part --------------<br>A non-text attachment was scrubbed...<br>Name: image.png<br>Type: image/png<br>Size: 48080 bytes<br>Desc: not available<br>URL: <<a href="https://protect2.fireeye.com/v1/url?k=5a238e22-05b8e42a-5a26ffac-ac1f6bdccbcc-ee06a4af1da35812&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2Fmailman.ccsds.org%2Fpipermail%2Fsis-dtn%2Fattachments%2F20240215%2F70f735de%2Fattachment.png" target="_blank">https://protect2.fireeye.com/v1/url?k=5a238e22-05b8e42a-5a26ffac-ac1f6bdccbcc-ee06a4af1da35812&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2Fmailman.ccsds.org%2Fpipermail%2Fsis-dtn%2Fattachments%2F20240215%2F70f735de%2Fattachment.png</a>><br>-------------- next part --------------<br>A non-text attachment was scrubbed...<br>Name: image.png<br>Type: image/png<br>Size: 79905 bytes<br>Desc: not available<br>URL: <<a href="https://protect2.fireeye.com/v1/url?k=c83cc342-97a7a94a-c839b2cc-ac1f6bdccbcc-17dd8d887bfde0f4&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2Fmailman.ccsds.org%2Fpipermail%2Fsis-dtn%2Fattachments%2F20240215%2F70f735de%2Fattachment-0001.png" target="_blank">https://protect2.fireeye.com/v1/url?k=c83cc342-97a7a94a-c839b2cc-ac1f6bdccbcc-17dd8d887bfde0f4&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=http%3A%2F%2Fmailman.ccsds.org%2Fpipermail%2Fsis-dtn%2Fattachments%2F20240215%2F70f735de%2Fattachment-0001.png</a>><br><br>------------------------------<br><br>Subject: Digest Footer<br><br>_______________________________________________<br>SIS-DTN mailing list<br><a href="mailto:SIS-DTN@mailman.ccsds.org" target="_blank">SIS-DTN@mailman.ccsds.org</a><br><a href="https://protect2.fireeye.com/v1/url?k=ac25f3ab-f3be99a3-ac208225-ac1f6bdccbcc-2472f80e5d5dbacd&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=https%3A%2F%2Fmailman.ccsds.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsis-dtn" target="_blank">https://protect2.fireeye.com/v1/url?k=ac25f3ab-f3be99a3-ac208225-ac1f6bdccbcc-2472f80e5d5dbacd&q=1&e=3ae2454f-c7be-47fb-9265-5a883d8026a6&u=https%3A%2F%2Fmailman.ccsds.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsis-dtn</a><br><br><br>------------------------------<br><br>End of SIS-DTN Digest, Vol 152, Issue 3<br>***************************************</span><o:p></o:p></p></div></div></div></blockquote></div></div></div></div></blockquote></div></div></body></html>