[Sis-csi] Text for Transport Layer Section

[Keith Hogie]

Keith Scott,

   I am still concerned that this misses what most missions will be
looking for and may do more to chase off users than attract them.
If we are trying to introduce concepts of using Internet protocols
to new missions I think we need to talk more to things they
understand and feel comfortable with.  The current contents
seems to go into lots more detail on concepts like "overlay
networks" and DTN.  These are concepts that will not apply to
any missions for many years to come.  They may be of some use
15 years from now when there is more of a mesh network available
in space but right now they are still very experimental.

See more specific comments below.  I can get you an alternate
version of this section next week if you are interested.

Keith Hogie


Scott, Keith L. wrote:
> Below is the new text for the transport layer section that includes UDP 
> and a generalization/expansion of the overlay section.  I am in the 
> process of uploading a new version that includes Howie's security 
> changes (thanks!) to CWE.
>         --keith
>  
> 
> 
>     1.1         Transport Layer
> 
> The standard transport protocols used with the Internet protocol suite 
> are TCP, which provides a reliable bytestream service, and UDP, which 
> provides an unreliable datagram service. Other services such as overlay 
> services that provide reliability without the requirement for 
> bidirectional end-to-end paths, or that provide reliable multicast, can 
> be built on top of TCP and UDP.
> 
> 
>       1.1.1        UDP
> 
> UDP provides an unreliable data delivery service comparable to standard 
> TDM and CCSDS packet delivery systems currently used for space 
> communication.  UDP packets may be lost or duplicated in the network, 
> and no feedback of such events is provided to the sender.  Because UDP 
> does not implement reliability or require any signaling from the 
> recipient to the sender, it can function over paths with arbitrary 
> delays and/or simplex paths.  UDP is commonly used for data delivery 
> where completeness is not required, such as cyclic telemetry.  If a UDP 
> packet containing one or a set of telemetry measurements is lost, it may 
> be enough to simply wait for the next packet, which will contain more 
> up-to-date information.

This starts out OK but then shifts into making it sound like you
can't or don't want to try doing reliable delivery with UDP.  If you
look at your own laptop you will find lots of things using UDP all
the time.  Actually, before you use TCP for most any activity your
laptop starts by using UDP to do a DNS lookup.  If you don't get
a "reliable" UDP response then your TCP connection never even
get started.

You probably also have NTP running over UDP to keep your clock on time.
Also any audio or video streaming you use is over UDP.  Most network
management systems use SNMP to reliably manage thousands of devices
and that's all over UDP.

For reliable file transfers, NFS was developed over UDP to allow
better adjustment of retransmissions mechanisms.  The MDP (predecessor
to NORM) file transfer protocol has been used for years by groups like
the US Postal Service and Kmart to reliably distribute files to
thousands of users using UDP over IP multicast over the open Internet.
We have also heard about the DMC satellites all using a UDP based
file transfer protocol to move their image files over 8 Mbps downlinks.

The main message is that UDP is used for all sorts of packet
delivery and file delivery applications and those are exactly the
sort of things that missions want to do.

> 
> While it is possible to use UDP for data transport and to implement 
> reliability at the application layer, care should be taken in doing so, 
> especially in a network that concurrently carries other Internet 
> protocol traffic such as TCP.  Applications using UDP would need (in a 
> mixed network) to ensure that they did not congest the network, either 
> by implementing some sort of congestion control mechanism or by careful 
> management of all link volumes.  Note that this is not considered a 
> problem if the application sends small amounts of data such as small 
> telemetry samples.  It becomes an issue only when when a UDP-based 
> application might want to send large amounts of data that could, if sent 
> all at once, overwhelm a router in the middle of the network.  The IETF 
> is currently investigating a datagram congestion control protocol  
> (DCCP) for such applications, though DCCP requires bidirectional 
> communications.
> 
> 

This whole paragraph seems to focus on "congestion" as such a major
issue.  Actually, TCP creates "congestion" issues on current NASA
operational networks.  The UDP data flows are limited by their RF
link bandwidths.  However, users running TCP based applications
over operational networks can cause much worse peaks in traffic.
As TCP starts up, it ramps up its data rate until it realizes it
needs to ease off due to congestion.  In the process it may
actually interfere with some constant rate UDP flows.

Also, congestion is something that missions and their flight software
have always dealt with.  Spacecraft have multiple possible streams
of data and their RF downlink is slower than their onboard data bus.
The flight software and hardware have been designed to make sure
that each data flow only uses its allocated about of bandwidth.

When we connect a fast PC to the Internet with a gigabit Ethernet,
then we do create a potential for creating congestion.  But then
even if you have a gig Ethernet connection to your PC, there is
probably a slower link leaving your building.  So if you try to
flood UDP data at 1 Gbps, most of it will get dropped before
getting out of your building.

When space missions get designed, the end-to-end links are all
considered, and packet data rates get set to make sure that the
mission doesn't clog up the links it is using.  For the next 5-10
years missions will continue to be carefully designed to make
sure they fully utilize their links and don't congest them.
Also, any missions with one-way links or very long delays will
always have to control their sending data rate since they cannot
get any TCP-like feedback for rate control information.

At some point many years in the future we may actually have
multiple end nodes flowing data over space routers and sharing
space links.  But for now we need to focus on how to get basic
infrastructure in place and get missions to start using basic
UDP/TCP/IP capabilities that they are comfortable with.

>       1.1.2        TCP
> 
> TCP provides a reliable, in-order bytstream delivery service without 
> duplication.  This means that when applications send data using TCP, the 
> sending TCP endpoint will attempt to detect lost data and will 
> retransmit data until it is acknowledged by the receiver.  TCP also 
> provides congestion control to attempt to keep from overloading the 
> network with too much traffic.  Because of the way reliability and 
> congestion control are implmeneted within the TCP protocol, TCP 
> performance can suffer in stressed environments characterized by large 
> bandwidth*delay products, high bit error rates, and significant 
> asymmetries in data rate.  The round trip light time from the Earth to 
> the Moon is on the order of 3 seconds, and an overall round trip time 
> including intermediate relays of on the order of 5 seconds will probably 
> be more typical.  This is enough to cause degradation in TCP 
> performance, especially if 'stock' end systems are used.
> 



> In the 1990s, CCSDS developed the Space Communications Protocol 
> Standards Transport Protocol (SCPS-TP) extensions to TCP to attempt to 
> extend the operating range over which TCP can perform efficiently.  
> While SCPS-TP provides a compatible application programming interface to 
> TCP, deploying the SCPS-TP extensions in every end host is often 
> impractical.  Instead, Performance Enhancing Proxies, or PEPs, are often 
> used to isolate the high bandwidth*delay links that can lower TCP 
> performance.
> 

At this point it seems that UDP has been discouraged, standard TCP
has problems and all future missions need to get into PEPs, overlay
networks, and eventually DTN.  Using IP is starting to sound real
complex and risky for new missions and they may stop reading and
go back to their traditional ways.


> 
>         1.1.2.1       Using Performance Enhancing Proxies to Improve TCP
>         Performance
> 
> Some of the performance problems of end-to-end TCP can be ameliorated 
> with the use of performance enhancing proxies, or PEPs.  For TCP 
> traffic, a PEP is a device that is in the network but that interacts 
> with the end-to-end TCP flow in order to improve its performance.  There 
> are a number of different kinds of PEPs discussed in [RFC3135], but one 
> of the most common types are split-connection PEPs.  Split connection 
> PEPs break the TCP connection into multiple connections, with the 
> connections across the stressed portions of the network using 
> technologies that are specifically designed and/or tuned for performance.
> 

Current missions like CHIPSat use standard FTP/TCP and work fine.  They
do their connection splitting by having the spacecraft first move a file
from the spacecraft to a FTP server at the ground station and later, in 
a separate TCP connection, the file gets moved to the control center.
The main thing is that the space link TCP connection is not disturbed by
any ground Internet congestion.  It is running over its fixed rate,
private, point-to-point RF link as designed.

Many current and future spacecraft will continue to use this simple
approach to avoid end-to-end TCP complications.  It's really about
determining where the "end-points" for any particular connection are.
In one scenario the "ends" would actually be the instrument in space
and the scientist on the ground.  However, since we don't have 24 hr.
end-to-end connectivity, there are actually multiple "ends" as data
moves in a store-and-forward fashion from instrument to scientist.
1 - data from instrument to onboard data store with some protocol
2 - later, files move from spacecraft to ground file store
3 - files move from ground storage to control center, LZP, or scientist
These scenarios can be either TCP or UDP based on mission
designers preferences and link characteristics.  Or UDP on some hops
and TCP on other hops.  That's what mission communication design
is about.

There are also cases where the RF downlink is actually faster than
the ground network links.  When the DMCs do their 8 Mbps dumps, the
links out of the ground stations are not that fast so the data
must be stored and forwarded later at lower rates.  Many NASA
satellites also do this with 75 or 150 Mbps dumps to stations
that don't have ground network links anywhere near that fast.

My point is that we have been dealing with intermittent links,
long delay links, noisy links, and non END-to-END links forever.
We can use UDP/IP in place of traditional frames and packets
in all these scenarios and we can use TCP/IP in carefully
selected scenarios.  We don't need to go pushing missions
into more exotic things like PEPs, overlay networks, and DTN
to support their basic packet and file delivery needs.  Right
now we need to focus on showing missions simple ways to use
UDP/TCP/IP to do their traditional data delivery operations.
If we ever get that in place, there will still be plenty of
time to add fancier features it we ever need them.


>  
> 
> *Figure **7: Split-connection PEPs break TCP connections into three parts.*
> 
> Figure 7 illustrates a pair of split-connection PEPs bracketing a 
> stressed link.  The PEP on the left terminates the TCP connection from 
> the left-hand host, and uses a separate transport connection (in this 
> case, SCPS-TP)
> 
> Note that in order to terminate TCP connections, the PEPs must be able 
> to see and modify the TCP headers.  This requires that the TCP headers 
> be 'in the clear' as they pass through the PEP, and not encrypted.  
> Network security mechanisms such as IP security (IPSEC) encrypt the 
> transport (TCP) headers, preventing the use of performance enhancing 
> proxies.  It is worth noting that most PEPs will pass IPSEC traffic, but 
> it will not benefit from the PEP's enhancement.  This means that IPSEC 
> can still be used if the security benefits it provides override the 
> performance degradation.
> 
> It is also worth mentioning that most of the benefits of IPSEC can be 
> obtained from transport layer security (TLS) mechanisms.  TLS encrypts 
> the payload user data at the application/transport layer boundary, 
> leaving the transport layer headers in the clear.  This allows PEPs in 
> the middle of the path to do their jobs by manipulating the headers 
> and/or terminating TCP connections.

As mentioned above, current missions already do their flavor of
"connection splitting" by using simple store-and-forward concepts.

> 
> 
>       1.1.3        Overlay Network Services
> 
> While TCP and UDP provide enough services to cover most terrestrial 
> communications needs, there are times when neither is in itself 
> particularly well-suited to an environment or application.  Perhaps the 
> two most common situations that require more support are reliable 
> multicast and communication when no end-to-end path exists.  TCP's 
> control loops that provide reliability and congestion control are 
> necessarily peer relationships between a single sender and a single 
> receiver.  Thus TCP is not suited to multicast traffic.  While UDP can 
> support multicast traffic, it does not provide any reliability or 
> congestion control.  Finally, both TCP and UDP rely on IP.  IP assumes 
> that network paths run uninterrupted from sender to receiver.  While 
> this is a good assumption in most terrestrial environments, it may not 
> hold for space applications, as spacecraft pointing, antenna/comm. 
> scheduling, and obscurations may conspire to interrupt communications. 

As mentioned above, MDP has been doing reliable file transfers using
UDP/IP multicast for years with thousands of simultaneous users.

Also the comment about IP assuming uninterrupted paths from sender to
receiver needs to be careful about identifying the "ends" as mentioned
earlier.  If the "ends" are the instrument and scientist, then you
won't normally have an end-to-end path.  But if you add in
intermediate store-and-forward "ends" IP works just fine.

In fact when I send this email, I'll bet that I don't have an
end-to-end IP path all the way from my computer to yours.  That
does not mean that I can't use IP to send my email.  What happens
is that I add some application level information (email addresses)
and send my message over a single hop to my postoffice.  Then
that postoffice uses the high level information to forward
my email over other one or more end-to-end IP connections
from postoffice to postoffice.  The messages end up at
destination postoffices and whenever you connect your computer
you can collect the message.

I can't support HTTP or SSH connections over this type of
file store-and-forward environment but science satellites
don't care about that.  Their primary goal is to collect
data and eventually move it to its destinations.  This can
and is being done today using simple TCP and UDP techniques
over IP on space links.

> 
> The common approach to providing enhanced services such as reliable 
> multicast or communication without an end-to-end path is to create a new 
> layer of protocol on top of either TCP or UDP.  This new layer of 
> protocol defines an overlay network as shown in Figure 8.  It may be the 
> case that only the end systems, some nodes in the network (as shown in 
> the figure), or all nodes implement the overlay protocol.  Nodes in the 
> overlay then use TCP, UDP, or data-link level communications to exchange 
> data.  The overlay may provide a reliable file replication service, a 
> reliable (unicast) file delivery service over intermittently-connected 
> links, or it may look like a transport protocol itself.
> 
>  
Is email considered an "overlay network" and what's wrong with
using it with some possible substitutions for message transfer
agent protocols to avoid TCP based issues.  I think the Mars folks
at Devon Island have already run tests of this approach.  Email is
something that missions can relate to much better than some abstract
concept like "overlay networks".

> 
> Figure 8: An overlay network (larger, dark circles) sparsely deployed in 
> an underlying network (smaller, white circles).
> 
> The Asynchronous Layered Coding (ALC) protocol [REF_RFC3450] forms the 
> basis for a number of overlay protocols, including Nack-Oriented 
> Reliable Multicast (NORM) [REF_RFC3940], a general-purpose reliable data 
> distribution protocol, and File Delivery over Unidirectional Transport 
> (FLUTE) [REF_RFC3926], a file delivery protocol.
> 
> The CCSDS File Delivery Protocol (CFDP) with its store-and-forward 
> overlay (SFO) procedures also implements an overlay network focused on 
> file delivery.  CFDP can run over TCP or UDP, or can be configured to 
> run directly over data link protocols such as AOS and Prox-1.

ALC, FLUTE, NORM, CFDP, etc. are all UDP-based and are the
types of simple protocols that can work well on space links.
These relate much more closely to how missions operate today.

The discussion about CFDP over TCP really seems to be gross
overkill and overhead.  If CFDP really can't handle out of order
packets then it seems that CFDP or its software implementation
needs fixing.  As Dai mentioned, delayed ACKs and retransmissions
are really the ultimate out-of-order packet arrival case.

Another issue that doesn't seem to be addressed is how these
various protocols affect things like memory and non-volatile
storage on space systems.  When you use TCP, PEPs, and DTN you
need to add more memory and storage on space nodes and that is
not at all popular with missions.  When you use UDP-based
file transfer protocols the file itself is your retransmission
window.  You don't need to keep data in memory for retransmission
since you can always go back to the sending application and
ask for a specific chunk of data again.  When you try to add
retransmission mechanisms like TCP and make things look
transparent to applications, then the intermediate layer
must buffer data and as delays increase the amount of
buffering increases.  With simple UDP file transfers you
don't need to add any memory as delays increase because
the original file is always sitting there.  UDP-based file
delivery mechanisms are highly independent of delays,
data rates, and file sizes.
> 
> A slightly different type of overlay network that is Delay/Disruption 
> Tolerant Networking (DTN) [REF_XXXXX].  DTN provides an 
> optionally-reliable datagram delivery service to applications, 
> regardless of whether end-to-end paths exist or not.  Reliable message 
> delivery is accomplished by a sequence of custody transfers from node to 
> node in the overlay rather than with end-to-end reliability as with 
> TCP.  Custody transfers are a function of the overlay protocol and don’t 
> depend on bidirectional paths between overlay nodes.  Thus a DTN node 
> might transmit a message on Tuesday using UDP over AOS and receive an 
> indication that some other node has taken custody of the message on 
> Wednesday, with that indication coming by way of a TCP connection over 
> Prox-1.
> 
>  
> 
> Unlike the overlays above, DTN is designed to accommodate changing 
> connectivity in addition to intermittency.  The DTN overlay is designed 
> to run its own routing protocol(s) independent of the underlying 
> network.  These DTN routing protocols can account for things the 
> underlying network does not, such as scheduled future periods of 
> connectivity.  Thus a DTN node might decide to break a message it is 
> currently forwarding into two parts, one to be sent now over UDP and 
> another to be sent over a future scheduled Prox-1 connection.  The 
> various pieces are then reassembled at the destination (or can be 
> reassembled at another intermediate node if they happen to meet).
> 
> To illustrate how overlay services can improve performance in 
> intermittently-connected environments, Figure 9 shows two views of a 
> notional four-hop network path.  The top view uses end-to-end networking 
> such as IP between the source at the top and the destination at the 
> bottom.  Time in the figure progresses to the right, and up/down 
> timelines for each link are shown.  A heavy bar centered on the thin 
> line for a link indicates that a particular link is up at a particular 
> time, and a thin line without a bar indicates that the link is down.  
> Data is represented by the heavy boxes that are above link connectivity 
> indicators, and the source is assumed to always have data to send.
> 
>  
> 
> *Figure **9: End-to-end networking requires a full path between source 
> and destination before any data can be sent.  A long-term 
> store-and-forward system can use individual links as they are available.*
> 
> In the end-to-end (top) portion of the figure, the source has to wait 
> until there is a complete path to the destination before any data can be 
> sent, which increases the latency and reduces throughput.  The 
> message-based store-and-forward system, on the other hand, gets the 
> first bit to the destination much faster, and has a higher overall 
> throughput.
> 
We have always done this sort of store-and-forward data delivery to
deal with intermittent links.  It's possible that a more elaborate
solution than something like basic email might be needed in the
future but that's still be be determined.  Right now DTN is still
very experimental and needs lots more use and real space based
investigation before we go pushing it as a solution for everything.

I still remember how the Xpress Transport Protocol (XTP) was
going to provide better performance, reliable multicast, and
all sorts of other neat features.  However, after a few years
of serious development and even forming a company to implement
it in silicon (Protocol Engines, Inc.) it became clear that it was
not going to be able to meet all its grand visions and be able
to scale up for large scale deployment.  DTN also needs more time
to mature and to do a better job of defining how it provides
significant benefits for space communication that can't be
achieved by much simpler approaches.
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Sis-CSI mailing list
> Sis-CSI at mailman.ccsds.org
> http://mailman.ccsds.org/cgi-bin/mailman/listinfo/sis-csi


-- 
----------------------------------------------------------------------
   Keith Hogie                   e-mail: Keith.Hogie at gsfc.nasa.gov
   Computer Sciences Corp.       office: 301-794-2999  fax: 301-794-9480
   7700 Hubble Dr.
   Lanham-Seabrook, MD 20706  USA        301-286-3203 @ NASA/Goddard
----------------------------------------------------------------------