[Moims-sc] Binary encoding of lists
Serge Lacourte
serge.lacourte at scalagent.com
Tue Oct 27 10:07:49 EDT 2026
Hi all,
I address 2 points in this answer to Stefan : Abstract or Concrete List
types, and type preservation
*_Abstract or Concrete List types_*
Stefan raises the question of the language mapping, however I believe
that the question I raised concerns MAL types and APIs. So I stay at the
MAL level in this mail.
The MAL defines a type model, and this model is used to define the
operation type signature, not just what will eventually circulate on the
wire. It shows in all layers of the MAL Architecture : at the service
layer (parameter types derived from the operation type signature), at
the MAL layer (exchanging MAL Messages built from MAL types), at the
transport layer (MAL Messages), and also at the encoding layer (MAL
Elements, even if this is less clearly specified).
Both Abstract and Concrete types may be used in the operation type
signature definition, however only Concrete types can be instanciated
and used at runtime (*1). When I suggests that Heterogeneous List is a
Concrete type, this allows me to have a consistent specification and to
apply this statement to all the lists. When you say that "There is no
“runtime Heterogeneous List”", it collides with this statement that only
Concrete types can be used at runtime (*2). With MAL v2 I can pass the
list (1, true) as a parameter for an operation which declared
List<Attribute> in its type signature. And this exists at the MAL level,
not just in a language mapping.
When I assemble all this together, the API of the the provider declares
a List<Attribute>, and the consumer must provide a concrete type to call
this API. The only concrete type that can hold (1, true) is the
Heterogeneous List. Of course this must be translated in a language
mapping, but this is another story. If I go back to the example of
Stefan at the end of the message, the case of a consumer providing an
IntegerList is a MAL level case, not restricted to any language mapping.
*_type preservation_*
In the first section I (hope that I) demonstrated that the MAL types are
part of the consumer and producer MAL API, and that an Heterogeneous
list is a Concrete type. The next step is:, does the MAL specify that
the Concrete type of the parameter received by the provider is exactly
the Concrete type of the parameter sent by the consumer (*3)? This is
not clear, and this is exactly the question raised by the last example
from Stefan. Olivier and I seem to vote yes for the type preservation,
i.e. the provider in Stefan's example should receive a List<Integer>. I
read Stefan's answer ambiguously, as he says on one side that "all MAL
type information" has been conveyed, however he allows the provider to
receive a List<Attribute>.
The MAL specification is ambiguous on this point, but I believe we
should discuss it and reach a common understanding. This impacts the
testbeds we are about to design. If the decoded object may be either a
List<Integer> or a List<Attribute>, then the testbed must accept both
answers. This may also impact the understanding of other MAL users. I
believe it is simpler to explain that the type information is preserved
(*4), as it matches the semantics of any existing programming language.
Best regards,
Serge
(*1) the MAL v2 specification says: "If a service specification
specifies a certain type as abstract and specifies its usage in a
service operation or in a Composite structure, then, at runtime, a
concrete type derived from the specified abstract shall be used.".
(*2) an alternative to keep the consistency of the specification,
without having Abstract or Concrete list types, would be to say that a
List is not a type, just a syntactic sugar to declare "an
arbitrary-length sequence of items". However this would add a lot of
complexity, and collide with other elements of the specification, such
as: "All Lists shall be extensions of the Fundamental Element type".
There is actually an implicit mapping between Concrete type and Short
Form Part, even if the specification only describes a one way relation:
"Each concrete type is defined with a numerical Short Form Part". All
lists of a Concrete type have their own SFP, and can be considered as
Concrete types. The last List object which can circulate on the wire is
the Heterogeneous List, so it has been given a specific SFP.
(*3) there is no clear answer to this question. As I said before, the
MAL APIs at all levels make reference to MAL types. It is natural to
assume that when the API of the transport is to convey a MAL Message
from one end to another end, then the Message will not be changed, and
this includes the types of the message elements. The closest element I
found, in MAL v1, is: "The specification of this for a particular
technology is called a Transport Specification. It defines the mapping
from the abstract MAL data structures into a specific and unambiguous
encoding of the messages for that specific data transport. (For further
information on this, see
reference [1].)".
(*4) it is preserved at the MAL level, but not all the type information
is perserved at the language mapping level. If the MAL operation
declares a List<Element>, the language mapping may allow to declare a
variable typed as List<Attribute>, and initialize it with the values 1
and true. The List<Attribute> type information would be lost, as the
provider would receive an Heterogeneous List, and set it to a
List<Element> typed variable.
Le 27/10/2026 à 11:05, Stefan.Gaertner at dlr.de a écrit :
>
> Hi all,
>
> I agree with Serge up to these two points, which I do not agree with:
>
> * an Heterogeneous List is a concrete type. It may hold elements of
> any MAL type. It is given 0 as SFP.
> * the statement 4.2.6.1.6 "If the type of the List is Abstract, then
> the List shall be a Heterogeneous List." is ambiguous. It could be
> interpreted as : If a service specification specifies a List type
> from an abstract element type as the parameter of a service
> operation, then, at runtime, an Heterogeneous List shall be used.
>
> There is no “runtime Heterogeneous List”. The distinction
> “Homogeneous” and “Heterogeneous” is purely at specification level. As
> such a “Heterogeneous List” is not a concrete type. It is a way of
> saying: We need a list here, whose elements may be of different type.
> What MAL does wrong here in my opinion, is to assign an SFP of 0. We
> should have left that out, just as for any other abstract type.
>
> Considering that “Homogeneous” and “Heterogeneous” are only valid
> concepts for declared types, not actual types, I also do not agree
> with Olivier’s wording “encode it as heterogeneous/homogeneous list”
> and “so the transport will have changed the MAL object, which is bad”.
>
> Instead of saying that I would make it very clear along the lines
> similar to the following:
>
> * A list declared as homogeneous is encoded as “single-type” list.
> * A list declared as heterogeneous is encoded as “multi-type” list.
> o Optimization: If all elements of a “multi-type” list are of
> same type, the list may be encoded as “single-type” list.
> o (For completeness: There are also “half-way” optimizations if
> only the area is the same for all list element types – but
> this does not add anything to the discussion.)
>
> Don’t nail me down on the words “single-type” and “multi-type”, which
> I just made up. These two words describe the concrete runtime data
> structures that need to go on the wire, and as such are both concrete.
> No types are changed: Although a homogeneous list encoded as
> “single-type” list looks the same as a “heterogeneous list” encoded as
> “single-type” list on the wire, the decoder can always tell these
> apart by using the context information. So yes, of course the decoder
> needs to know what structure to decode from the specification! The
> binary encoding is not self-describing and never was. This is already
> true for a simple String: If you receive [2, 72, 73] there is no way
> to know if you have to decode a List<Integer> with [72, 73] as
> elements or the String “HI” without having context information (the
> specification) available.
>
> Now, one thing I believe Serge was hinting at: What does this mean in
> a concrete programming language mapping?
>
> A decoder might decode single- and multiple-type lists to different
> programming language types. But this is implementation-dependent. We
> don’t care if provider and consumer use the same programming language
> types on both sides. What we care about is that both sides know
> exactly what MAL types they work with – and this is fixed by
> specification plus conveying of type information on the wire where needed.
>
> As an example: Suppose my encoder accepts the programming language
> object “IntegerList [1, 9, 0]“ for a parameter defined as
> “List<Attribute>”. I have an efficient encoder, so this gets encoded
> as single-type list on the wire. On the other side, I have a dumb
> decoder that knows to expect a List<Attribute> and as such
> instantiates a programming language object “AttributeList [1, 9, 0]”.
> Now you might say: “But IntegerList is a different type than
> AttributeList, serialization changed my type!” But did it? On both
> sides you still know that your MAL type is “List<Attribute> [Integer,
> Integer, Integer]”. It is not the responsibility of MAL to convey the
> exact programming language type information. In cases of different
> programming languages this usually is not even possible. It is enough
> to simply convey all MAL type information, and this we made sure.
>
> Cheers,
>
> Stefan
>
> *From:*MOIMS-SC <moims-sc-bounces at mailman.ccsds.org> *On Behalf Of
> *Churlaud Olivier via MOIMS-SC
> *Sent:* Montag, 26. Oktober 2026 19:05
> *To:* LACOURTE Serge (contact) <serge.lacourte at scalagent.com>;
> moims-sc at mailman.ccsds.org
> *Subject:* Re: [Moims-sc] Binary encoding of lists
>
> Hi all,
>
> I’ll try to summarize the issue with an example, and what would be my
> decision. As usual, feel free to disagree J
>
> *_Description of the issue_*
>
> Entity A exchanges data with Entity B.
>
> Entity A sends a MAL List<Attribute> to Entity B.
>
> In a very particular exchange, it happens that all Attributes are
> Integers, for instance: [1, 9, 0]
>
> With the optimization that Stefan proposes, Entity A can :
>
> 1. either encode it as an heterogenous list, which has the short form
> part 0: List<Attribute> [ <int> 1, <int> 9, <int> 0] or a
> 2. or encode it as an homogeneous list, which would have the short
> form part -11 : List<int> [1, 9, 0]
>
> *_Why it’s an issue_*
>
> __
>
> In case 1), Entity B will indeed receive a List<Attribute> (short form
> part 0), whereas in case 2, Entity B will receive a List<Integer>
> (short form part -11), so the transport will have changed the MAL
> object, which is _bad._
>
> On the other hand, we said that Entity B knows which operation is
> being used, and therefore the expected data. So Entity B knows
> beforehand if the data is a List<int> or a List<Attributes>, so can do
> the type casting while decoding the message.
>
> *_The decision_*
>
> The point here is to decide between 2 options: can we assume that the
> data type is known before hand, and that the consumer uses this
> information to do the right type castings or not?
>
> *_My conclusion_*
>
> In my opinion, it should be authorized, otherwise we cannot do this
> kind of optimization.
>
> The consequence is that the encoding of lists must have two decoding
> paths:
>
> * an heterogeneous list which all elements have the same type can be
> encoded
> o as an heterogeneous list (short form part = 0, and short form
> part of each element repeated)
> o or has an heterogeneous list (short form part = 0, and short
> form part of each element repeated): it’s an optimization and
> it’s legal
> * An homogeneous list can be encoded
> o as an homogeneous list (short form part = -11 for integers)
> o or as an heterogeneous list (short form part = 0, and short
> form part of each element repeated): it’s stupid but legal
>
> * the decoders should be compliant of that.
>
> Since MAL is ambiguous on this, the transport should be prescriptive:
> this should be described in the MAL SPP/Binary Binding. For HTTP/XML,
> we should either say the same thing, or forbid it explicitly (since we
> don’t need optimization here, I would go for the simplest in terms of
> book changes).
>
> I hope the point is clearer now. Let’s discuss it next Monday J
>
> Cheers
>
> Olivier
>
> *De :*MOIMS-SC <moims-sc-bounces at mailman.ccsds.org> *De la part de*
> Serge Lacourte via MOIMS-SC
> *Envoyé :* vendredi 23 octobre 2026 11:17
> *À :* moims-sc at mailman.ccsds.org
> *Objet :* [Moims-sc] Binary encoding of lists
>
> *EMAIL EXTERNE :*Cet e-mail provient d'un expéditeur externe. Faites
> preuve de vigilance avant de cliquer sur un lien ou d'ouvrir une pièce
> jointe. Vérifiez que l'expediteur et le contenu du message sont légitimes.
>
> Dear WG members,
>
> we discussed several optimizations proposed by Stefan for the binary
> encoding. One of them concerns the encoding of lists. I understood the
> proposal as the possible encoding of a list holding elements of a
> single concrete type with a single encoding of the type instead of
> repeating the encoding of this type for all the elements of the list.
> This bothered me, and I try to discuss it more formally here.
>
> I explain in the following my understanding of the MAL requirements. I
> would like to have your opinion on this.
>
> There is no formal API of an encoding. An informal API of the encoding
> could be to convert a MAL data into an array of bytes(*1), and the
> reverse. I would go one step further, asserting that the decoding of
> the array of bytes which results from the encoding of a MAL data
> should be the exact copy of the original MAL data, including its exact
> MAL type.
>
> With MAL v1 this was clear and obvious. However MALv2 introduces grey
> cases that have probably not been cleared enough, and I believe that
> some ambiguity may remain. The heterogeneous list is such a case.
>
> The MAL specifies Abstract and Concrete types. An operation may
> declare an Abstract type in an operation parameter. At runtime, this
> parameter will be filled with a Concrete type that "derives" from the
> Abstract type. The "derivation" rules is clear for Composite types, it
> is less clear for List types. The MAL specification never tells the
> Abstract or Concrete type of a List. It does define List as a way to
> define a MAL type from the Abstract or Concrete type of the list
> elements. Il also defines the concepts of Homogeneous and
> Heterogeneous lists, both concepts being assigned a Short Form Part.
> It never formally defines what List types are Abstract or Concrete. My
> understanding is the following :
>
> * a List of Concrete element type is a Concrete type. Its Short Form
> Part is the negative value of the SFP of the element type.
> * an Homogeneous List is a global naming for all the lists defined
> from a Concrete element type
> * a List of Abstract element type is an Abstract type
> * an Heterogeneous List is a concrete type. It may hold elements of
> any MAL type. It is given 0 as SFP.
> * the statement 4.2.6.1.6 "If the type of the List is Abstract, then
> the List shall be a Heterogeneous List." is ambiguous. It could be
> interpreted as : If a service specification specifies a List type
> from an abstract element type as the parameter of a service
> operation, then, at runtime, an Heterogeneous List shall be used.
>
> However this interpretation of statement 4.2.6.1.6 is clearly too
> restrictive. If concrete composite type C extends abstract composite
> type A, then List<C> can be used in place of a List<A> declared
> parameter. And list<C> is an Homegeneous List. There are then 2
> options matching a List<A> declared parameter :
>
> 1. Homogeneous List<C>, where C extends A
> 2. Heterogeneous List holding only elements which extend A (runtime
> check)
>
> In the case 1 above, the parameter will obviously be decoded as a
> List<C>. The question we have to answer now, which is not considered
> in the MAL specification, is : in the case 2 above, where an
> heterogeneous list is provided, should it always be decoded as an
> heterogeneous list, even if all the enclosed elements are of the same
> type C? My answer is yes, it should. This is why I reacted to the
> encoding proposal to allow encoding it as a List<C>. I agree that this
> optimization is still possible, as long as we agree that the parameter
> is eventually converted into an heterogeneous list. This is possible
> as the encoder knows the declared type of the operation.
>
> Serge
>
> (*1) with the split binary, this is not exactly true. It is true for
> the overall encoding of the MAL message, however the encoding usually
> refers to the encoding of each individual MAL data type in the MAL
> message body. The encoding of the MAL header is defined in the
> transport, with a specific encoding which may not follow the encoding
> of the body. So the split binary encoding transforms a MAL data into 2
> separate arrays of bytes, one for the included booleans, and another
> for all other included data types.
>
> --
> Serge Lacourte
> Directeur general
> ScalAgent Distributed Technologies SA
> tel. +33 4 76 29 79 81
> mobile. +33 6 86 47 41 06
--
Serge Lacourte
Directeur general
ScalAgent Distributed Technologies SA
tel. +33 4 76 29 79 81
mobile. +33 6 86 47 41 06
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ccsds.org/pipermail/moims-sc/attachments/20261027/5406ff40/attachment-0001.htm>
More information about the MOIMS-SC
mailing list