[phpxmlrpc] xmlrpc_encode_entitites causing parse error

Tue Nov 15 17:57:27 GMT 2005

On Nov 15, 2005, at 11:31 AM, Gaetano Giunta wrote:

> Very toughtful response.

Man, I love cross-linguistic typos...makes great new English words:  
"toughtful" = "tough thoughtfulness". Brilliant.

> UTF-8 everywhere is fine and dandy but for 2 aspects:
>
> - in fact XML-over-http without a charset declaration SHOULD be  
> assumed to be ISO-8859-1 (there is a RFC somewhere about that,  
> which I cannot recall now).

Hmmm. The XML 1.0 spec (http://www.w3.org/TR/2000/REC-xml-20001006)  
reads:

Because each XML entity not accompanied by external encoding  
information and not in UTF-8 or UTF-16 encoding MUST begin with an  
XML encoding declaration, in which the first characters must be '<? 
xml', any conforming processor can detect, after two to four octets  
of input, which of the following cases apply.

RFC 2376, however, offers suggestions for XML MIME-types sent over  
HTTP, but it reads (pardon the length):

Although listed as an optional parameter, the use of the charset
       parameter is STRONGLY RECOMMENDED, since this information can be
       used by XML processors to determine authoritatively the character
       encoding of the XML entity. The charset parameter can also be  
used
       to provide protocol-specific operations, such as charset-based
       content negotiation in HTTP.  "UTF-8" [RFC-2279] is the
       recommended value, representing the UTF-8 charset. UTF-8 is
       supported by all conforming XML processors [REC-XML].

       If the XML entity is transmitted via HTTP, which uses a MIME-like
       mechanism that is exempt from the restrictions on the text top-
       level type (see section 19.4.1 of HTTP 1.1 [RFC-2068]), "UTF-16"
       (Appendix C.3 of [UNICODE] and Amendment 1 of [ISO-10646]) is  
also
       recommended.  UTF-16 is supported by all conforming XML  
processors
       [REC-XML].  Since the handling of CR, LF and NUL for text  
types in
       most MIME applications would cause undesired transformations of
       individual octets in UTF-16 multi-octet characters, gateways from
       HTTP to these MIME applications MUST transform the XML entity  
from
       a text/xml; charset="utf-16" to application/xml;  
charset="utf-16".

       Conformant with [RFC-2046], if a text/xml entity is received with
       the charset parameter omitted, MIME processors and XML processors
       MUST use the default charset value of "us-ascii".  In cases where
       the XML entity is transmitted via HTTP, the default charset value
       is still "us-ascii".

...which implies that us-ascii, not iso-8859-1, is the default (but  
not really a problem if you're encoding everything outside of ASCII).  
But I know that my RDFParser class, for example, defaults to "utf-8"  
and overrides that only if the encoding is specified as something  
else in the xml delaration. I assume I made that decision for good  
reasons, though I don't remember them now!

Still, the number of factors affecting encoding and transmission are  
unbelievably complex. In my software, for example, there is:

1) Page encoding used when users submit data via a form (mine: UTF-8)
    a) Default charset header sent by Apache (mine:  UTF-8)
    b) Default charset set in META tags (mine: UTF-8)
    c) Charset setting of client browser (no control!)
2) Encoding of database (mine: MySQL 3.x, so limited to ISO-8859-1)
3) Encoding of page used to display data (Irrelevant to XML-RPC  
transfers, but 1a,1b,1c apply)
4) PHP internal encoding
5) XMLRPC library internal encoding
6) XML declaration charset (optional, but highly recommended by spec)
7) text/xml MIME type charset declaration (optional, mine: text/ 
xml;charset=utf-8)
8) application/xml MIME type charset declaration (optional)

...and since all of them could be set to different encodings, getting  
it all straight is a dizzying adventure. Add to that the complexity  
of handling things like users copying text from a Word document  
created in Windows-1252 and pasting into a form on a UTF-8 page,  
and...ugh! Sometimes I just want to kill myself.

While I suppose that attempting to convert all data into us-ascii  
through entity encoding gives us the "least common donominator"  
solution -- make everything 7-bit! -- it obviously isn't working  
perfectly. So perhaps any solution that simply makes it work,  
regardless of whether or not it changes the use of  
$xmlrpc_internalencoding, would be good. I did wonder about the  
utf8_encode() function, and why you didn't simply use that instead of  
$character = ("&#".strval($code).";"); Won't that do all the right  
work for you?

In any case, I think you should try to make the XMLRPC library follow  
as closely as possible the relevant spec/RFC "recommended" behavior,  
and let that be your guide.

> Adding some extra settings to client/server objects is fine, but  
> the causal user might not be used to using those, and backward  
> compatability is a primary concern to me.
> Traduced in code that would probably mean adding some hacky stuff  
> of the sort "object default charset preference is undefined, and  
> while still undefined use global variable, otherwise use object  
> preference" (doable but ugly).
> The though part is letting the client object communicate the  
> desired charset encoding to the xmlrpcval object, since the  
> responsibility of creating serialized content is left to the  
> xmlrpcval object itself (and I'm surely not changing that  
> fundamental assumption).

If you converted $xmlrpc_internalencoding to a property of xmlrpcmsg  
instead of a global variable, then you could simply set it to default  
to "iso-8859-1" in the constructor method for the class object. So  
you maintain your default, but allow users to reset it through  
scripting.

> ps: the real (only ?) advantage of using variables instead of  
> constnts for things such as internal_encoding is that you can  
> redefine them not inside the xmlrpc lib but just after its  
> inclusion, eg.
> <?php include('xmlrpc.inc'); $xmlrpc_internal_encoding = 'UTF-16';  
> echo 'etc...'; ?>
> this way you do not have to change anything when updating...

Ah, yes, this is true, and I hadn't really thought of such a simple  
thing (but the same method holds true for using an object property).

How the PEAR people are handling this:
http://pear.php.net/bugs/bug.php?id=52
["According to RFC 3023 section 3.1, the encoding specified in the <? 
xml encoding=... ?> tag should be ignored for XML received over HTTP  
in favor of the encoding specified in the Content-Type header (e.g.  
"Content-Type: text/xml; charset=iso-8859-1)."]

I found another developer reflecting on these same questions, for a  
blogging app that uses XML-RPC:
http://ecto.kung-foo.tv/archives/000975.php

Other messages about the default encoding of unspecified xml documents:
http://groups.yahoo.com/group/xml-rpc/message/45
http://mail.zope.org/pipermail/zope-collector-monitor/2004-October/ 
004361.html (in reverse chronological order)

Cheers,
spud.

-------------------------------------------------------------------
a.h.s. boy
spud(at)nothingness.org            "as yes is to if,love is to yes"
http://www.nothingness.org/
-------------------------------------------------------------------