2010-11-13

Avoiding URL length limits when loading pages with Javascript

Working on a web application that must work with IE, I encountered problems with opening windows (via window.open), or loading content in iframes, that required a large set of parameters to the URL provided. IE has a hard-coded limit on the size of URL strings, so parameter data could get clipped.

Also, it is not pretty to have a large URL string showing in the address bar.

Therefore, the solution was to use an HTTP POST based method.

My first attempt was to use AJAX with a POST request to retrieve the desired page, and then do a document.write() into a blank window. However, I discovered that IE does not process data in a serial manner via document.write(). For example, if the page content contains <script> elements referencing external resources to load, IE will not process those in sequential order, so if you have Javascript code that calls a function that is defined by a previous <script>, you can get a runtime error. IE will load the external javascript resources, but they are done asynchronously, with IE not waiting to execute subsequent Javascript in the page that comes after the <script>.

The solution I came up with is to create a transient page that performs a form post. I.e. In the blank window, I create dynamic HTML page that contains an HTML form with the set of parameter data defined as hidden fields. The HTML page created has an onLoad action to submit the form.

With this approach, the browser is directly fetching the resource, so the document.write() problem described earlier (for IE) is no longer a factor.

What follows is a function for loading a document using a POST-based method:

loadDocument = function(
/*Object*/args
)
{
// summary:
// Load a document for a given document node.
// description:
// This function uses a POST-style method for loading the
// contents of a document node, where the document requested
// takes an arbitrary number of request parameters. This
// function works-around limitations of some browsers where
// there is a limit to the length of query-string parameters
// for GET-based requests.
//
// args: Object
// Object contain the following properties:
//
// doc: Node
// Document node to load content for.
// url: String
// URL to fetch content from.
// params: Object?
// Properties of string values representing parameters
// for the request denoted by url.
// message: String?
// Message text (HTML) to display while content is
// loading.

var doc = args.doc;
doc.open();
doc.write('<html>');
doc.write('<script type="text/javascript">');
doc.write('function submitForm(){document.forms[0].submit();}');
doc.write('</script>');
doc.write('<body style="background-color:#FFF;" onLoad="submitForm()">');
if (args.message) {
doc.write(args.message);
}
doc.write('<form method="post" action="');
doc.write(args.url);
doc.write('">');
if (args.params) {
for (var p in args.params) {
var v = args.params[p];
if (v instanceof Array || typeof v == "array") {
for (var i=0; i < v.length; ++i) {
doc.write('<input type="hidden" name="');
doc.write(p);
doc.write('" value=\'');
doc.write(escapeHTML(v[i]));
doc.write('\'></input>');
}
} else {
doc.write('<input type="hidden" name="');
doc.write(p);
doc.write('" value=\'');
doc.write(escapeHTML(args.params[p]));
doc.write('\'></input>');
}
}
}
doc.write('</form></body></html>');
doc.close();
}

// Helper function to escape HTML specials for use above.
escapeHTML = function(/*String*/s) {
// summary: Convert HTML special characters to entity references.
// s: String to escape.
if (!(typeof s == "string" || s instanceof String)) {
s = s + ""; // force it to a string
}
return s.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;")
.replace(/"/g, "&quot;")
.replace(/'/g, "&#39;");
}


The primary function works on a Document object, allowing it to not just be used for new windows, but for any object that contains a Document object, like frames and iframes.

Simple example using the above:

var w = window.open("");
loadDocument({
doc: w.document,
url: "http://example.com/some/resource/",
params: {
p1: "Hello",
p2: "World!"
}
message: 'Loading..."
});

2010-01-31

HOW-TO: Posting to Gmail servers with non-SSL/TLS-aware MUAs

I'm a long-time user of the nmh mail user agent (MUA), <http://www.nongnu.org/nmh/>, and its predecessor MH before that.

It's an old MUA. It currently does not support submitting emails to mail servers that require SSL/TLS connections. Other venerable MUAs have the same problem. Fortunately, there is an OSS program that provides proxy capabilities that can negotiate the SSL/TLS part, and if necessary, the SMTP AUTH part if the MUA does not support it: DeleGate, <http://www.delegate.org/>.

DeleGate provides proxy capabilities for various Internet protocols (e.g. http, ftp), but for this article, all we care about is its SMTP capabilities.

Nice thing about DeleGate is you do not need root priviledges to run it. You can have it listen to local client requests on any port you choose and forward the request to remote server (which will be smtp.gmail.com in this article).

Best way to show how to use it is with an example:

The following starts delegated on the local system on port 25 (the standard SMTP port):

delegated -Plocalhost:20025 +=$HOME/delegate/conf/gmail.conf


This says to listen for client requests on port 20025 and to read additional configuration parameters from gmail.conf located under delagate/conf of your home directory (you can specify all parameters on command-line, but that gets kind of ugly). In my gmail.conf file, I have something like the following:

SERVER=smtp://smtp.gmail.com:587
STLS=fsv
AUTHORIZER=-list{localuser:MD5:964ba203464913531aa73b9f774b58f7}
MYAUTH=username@gmail.com:pass:smtp


SERVER specifies what remote server the proxy should connect to. In this case, we specify that is the SMTP server smtp.gmail.com on port 587 (TLS port).

STLS setting specifies that SSL/TLS should be used when connecting to remote server (smtp.gmail.com).

The AUTHORIZER parameter is not required, but if specified, client must (SASL) authenticate to the proxy. The username is "localuser" (change to whatever your want). This may be handy so other users on the local system cannot hijack your proxy server and post email thru your Gmail account. If you are the sole user of your system, client authentication may not be needed. Also, if your MUA does not support SASL, you should not set AUTHORIZER.

To determine if your MUA supports SASL, if your MUA provides the ability to specify an username and password when posting a message to a mail server, it probably supports SASL. If no such capability is provided, it probably does not.

The password listed for AUTHORIZER can be provided in plaintext, but DeleGate will show in its log the MD5 hash of it, which you can then insert into the configuration file. A plaintext version of AUTHORIZER is as follows:

AUTHORIZER=-list{localuser:password}


The MYAUTH specifies the username/password to use for authenticating to the remote server (smtp.gmail.com). This should be set to your gmail address and gmail password. Unfortunately, the password cannot be encrypted since DeleGate needs to send the password to the remote server. Therefore, make sure the configuration file is only readable by you and no one else. For Unix-based users, the chmod command can be used:

chmod 600 gmail.conf


For Windows users, right click on the file and select Properties. Then select the Security tab. Make sure you are the only user that has read access to the file.

With DeleGate running, just configure your MUA to post messages to localhost:2005 and you should be good to go.

NOTES:

If you cannot specify an alternate port number for posting messages, you will need to run DeleGate on port 25. If this is the case, you will need to have admin priviledges to start DeleGate. DeleGate does provide an option for it to drop priviledges to a separate user. See documentation for more details.

DeleGate supports multi-user access, but configuration becomes more complex. If you want to support multi-user access, see the documentation.

For nmh user, SASL is supported, so client-based connections can be restricted, especially if running DeleGate as a personal proxy. However, for nmh, if DeleGate is not running on port 25, the whom command at the "What Now?" will not work. The command will also not work if you enable SASL for client connections. For whom to work, nmh code modifications are required. I've committed changes into the nmh project so SASL support is available for the whom command. You'll need to check out the latest source and build if you need the feature.

2009-10-28

Converting XML Catalogs to TR9401 Catalogs

Having worked with SGML/XML for some time now, I still encounter software that does not support XML Catalogs (including software I wrote), but do support TR9401 Catalogs.

Instead of trying to update/modify older software, I found it easy to write an XSLT to convert an XML Catalog into a TR9401 catalog. The XSLT is viewable/downloadable from the following link: xmlcat-to-opencat.xsl

For features supported in XML Catalogs but not in TR9401, the XSLT will print out warnings, but continue processing.

Unsure if anyone else may find the transform useful, but it comes in handy when I need to use my old DTD parsing tools.

2009-08-17

Providing meaningful errors in dojo.xhr calls

I've been working with Dojo lately, and one problem is the useless error messages one gets back from the server, especially when using dojo.xhr. The problem is that the response body from the server is generally some large HTML message that may not be useful for display purposes in an AJAX-based calling environment.

To remedy this, I made some modifications to the server-side (implemented in Java servlets) to capture exceptions and return a JSON structure if the request is an AJAX request. This makes it easier for the Javascript code to process error text more easily.

On the client-side, I have some basic utility functions to facilitate capturing/reporting errors from dojo.xhr calls.

First, the server-side:

Before I Start:

Before providing detail, I should note that I've created a base servlet
class that all my servlets sub-class. This allows me to centralize
operations and services for all my servlets. In the context of this post,
this becomes useful for the purposes of exception handling. The base
class checks all exceptions from sub-classes allowing centralization
of error handling based upon the type of exception that occurred.

I also have a custom request object to encapsulate the HttpServletRequest
and HttpServletResponse objects. This encapsulation allows me to add
support for additional capabilities beyond the standard servlet API
related to request/response. For example, I provide file-upload support
and still provide a single consistent interface for servlets to access
request parameters.

Some of the code provided here may contain references to custom classes I
use, but I believe they are self-explanatory.

How to detect an AJAX request:

public boolean isAjaxRequest() {
String reqwith = req.getHeader("x-requested-with");
if (StringUtil.isBlank(reqwith)) return false;
if (reqwith.toLowerCase().equals("xmlhttprequest")) return true;
return false;
}

The above works for most cases, and wrt Dojo, definitely works
since Dojo sets the proper request headers.

Capturing Servlet Exceptions:

In my base servlet class, I catch all exceptions:


} catch (Exception e) {
if (r.isAjaxRequest()) {
log.error("Exception caught for AJAX request: "+e, e);
sendJSONError(resp, HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
e.getMessage());
} else {
if (e instanceof ServletException) {
throw (ServletException)e;
}
throw new ServletException(e.getMessage(), e);
}
}

I check if the request in an AJAX request, and if so, utilize the sendJSONError() method to do what I want. sendJSONError() looks like the following:

private void sendJSONError(
HttpServletResponse resp,
int status,
String message
) throws IOException {
Writer w = resp.getWriter();
resp.setStatus(status);
resp.setContentType("application/json");
w.write("{");
w.write("\"status\":");
w.write(Integer.toString(status));
w.write(",");
w.write("\"message\":");
w.write(JSONObject.quote(message));
w.write("}");
}


Capturing/display Errors in Dojo:

On the client-side, I have two utility functions. One is a generic error handling function for dojo.xhr that will popup an alert displaying the error message received from the server. The other is a function to extract the error message from the servlet response, mainly for use by custom error functions that need to do more than display an alert dialog:


mylib.xhrErrorAlert = function(
/*Object*/response,
/*Object*/ioArgs
)
{
var prefix = "Error: ";
if (ioArgs.args.myAlertPrefix) {
prefix = ioArgs.args.myAlertPrefix;
}
alert(prefix + mylib.getXhrResponseMessage(response));
}


mylib.getXhrResponseMessage = function(
/*Object*/response
)
{
var msg = response.message;
if (response.responseText != undefined &&
response.responseText.charAt(0) == '{') {
try {
var o = dojo.fromJson(response.responseText);
if (o.message != undefined) {
msg = o.message;
}
} catch (err) {
}
}
return msg;
}

The following is an example of how to use the alert function:

dojo.xhrPost({
url: url,
handleAs: 'json',
load: function(response, ioArgs) {
...
},
myAlertPrefix: 'Error doing something: ',
error: mylib.xhrErrorAlert
});

Note, there is a reliance that the message text is meaningful in the exception thrown on the server-side. Regardless, this provides a better error message than what is normally available in response.message, which normally contains the canned message the server defines for the given HTTP response code versus the message text from the underlying exception.

2009-08-16

Getting fixed-width font for Gmail messages

Gmail labs has a feature to display a message in a fixed-width font, but you have to manual select it from a menu each time you want.

The following is a quick CSS userContent.css hack for Firefox that will cause message content to always display in a fixed-width font:

@-moz-document domain(google.com) {
.ii {
font-family: monospace;
font-size: 75%;
}
}

I use "monospace" since it will use what I have specified in my font preferences in FF. The key item is the ".ii" class.

The only negative side-effect is that non-text messages, like HTML, will have the default font be monospace. However, I'm currently willing to live with that limitation.

I posted a suggestion on the Gmail labs group for Google to define CSS classes based upon content media-types, so CSS settings can be constrained for a given media-type. Who knows if they will ever do such a thing.

Disabling ads in Gmail

For those that do not know, Firefox supports the ability for a user to define their own custom CSS settings, allow the user to customize how pages appear in FF. The file is called userContent.css and is located under the chrome/ directory of your profile.

As an exercise, I tried to see if it is possible to suppress the Google ads displayed in the Gmail web interface. Well, with a little help from Firebug to examine the HTML/CSS structure of Gmail pages, I came up with the following:

.vb {
display: none;
}
.u5 {
display: none;
}
table[class~="Bs"] > tr > td:nth-of-type(+3) > div:nth-of-type(+2) {
width: auto !important;
}
table[class~="Bs"] > tr > td:nth-of-type(+3) > div:nth-of-type(+2) >
div > div:last-child
{
display: none;
}

The above works best in FF 3.5, especially since the last items utilize CSS3 selectors.

FF supports the ability to limit the settings for a given domain. I Include the above in the following block so it only applies to google.com sites:

@-moz-document domain(google.com) {
...
}