Debugging pydap and Apache's mod_wsgi

WSGI is supposed to be a loosely defined interface that allows you to plug an web application into any web server which implements WSGI. I had been operating on the naive assumption that this was completely true (as opposed to mostly true), and had a nasty surprise when we went to deploy our web application to Apache/mod_wsgi after developing with Paste.

Every single data request to pydap came back with an ugly, ugly 500 internal server error. Badness. A data portal is supposed to serve data, so that was pretty much critical functionality. The logs reported:

[Mon May 07 17:33:47 2012] [error] [client 142.104.194.101] mod_wsgi (pid=18882): Exception occurred processing WSGI script '/var/www/tools/data_portal/data_portal.wsgi'., referer: http://medusa.pcic.uvic.ca/tools/data_portal/auth/pydap/Baseline_Historical_Climate_Data/forcings_new_symap_BC_22AUG2011_1950-2006_t0.nc.html
[Mon May 07 17:33:47 2012] [error] [client 142.104.194.101] TypeError: sequence of byte string values expected, value of type unicode found, referer: http://medusa.pcic.uvic.ca/tools/data_portal/auth/pydap/Baseline_Historical_Climate_Data/forcings_new_symap_BC_22AUG2011_1950-2006_t0.nc.html

At least I got it to the point where some of the response would come back, so I had a place to go digging in the code:

Dataset {
    Grid {
        Array:

Response Headers

HTTP/1.1 200 OK
Date: Tue, 08 May 2012 00:33:47 GMT
Server: Apache/2.2.16 (Debian)
XDODS-Server: dods/2.0
Content-description: dods_ascii
Last-Modified: Mon, 19 Sep 2011 22:41:11 -0000
Access-Control-Allow-Origin: windy.pcic.uvic.ca
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=15, max=94
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain

Request Headers

GET /tools/data_portal/auth/pydap/Baseline_Historical_Climate_Data/forcings_new_symap_BC_22AUG2011_1950-2006_t0.nc.ascii?tasmax[0:1:0][0:1:186][0:1:399]& HTTP/1.1
Host: medusa.pcic.uvic.ca
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.3) Gecko/20100101 Firefox/10.0.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Referer: http://medusa.pcic.uvic.ca/tools/data_portal/auth/pydap/Baseline_Historical_Climate_Data/forcings_new_symap_BC_22AUG2011_1950-2006_t0.nc.html
Cookie: beaker.session.id=2dbac1bffdfc49d28080c4971a40879b

After tracking down the problem, apparently pydap was mixing up unicode objects and str object in the response. As a workaround, I wrote a patch to str()ify everything that was being yielded.

diff -r a2af8159a8dc pydap/responses/ascii.py
--- a/pydap/responses/ascii.py	Sat May 05 16:43:28 2012 -0300
+++ b/pydap/responses/ascii.py	Tue May 08 14:22:19 2012 -0700
@@ -20,11 +20,11 @@
     def serialize(dataset):
         # Generate DDS.
         for line in dds_dispatch(dataset):
-            yield line
+            yield str(line)
         yield 45 * '-'
         yield '\n'
         for line in dispatch(dataset):
-            yield line
+            yield str(line)
         if hasattr(dataset, 'close'): dataset.close()
 
 
@@ -44,7 +44,6 @@
             yield line
         yield '\n'
 
-
 def _sequence(var, printname):
     yield ', '.join(child.id for child in var.values())
     yield '\n'
diff -r a2af8159a8dc pydap/responses/dds.py
--- a/pydap/responses/dds.py	Sat May 05 16:43:28 2012 -0300
+++ b/pydap/responses/dds.py	Tue May 08 14:22:19 2012 -0700
@@ -32,13 +32,13 @@
 
 def structure_builder(name):
     def func(var, level=0):
-        yield '%s%s {\n' % (level * INDENT, name)
+        yield str('%s%s {\n' % (level * INDENT, name))
 
         # Get the DDS from stored variables.
         for child in var.walk():
             for line in dispatch(child, level=level+1):
-                yield line
-        yield '%s} %s;\n' % (level * INDENT, var.name)
+                yield str(line)
+        yield str('%s} %s;\n' % (level * INDENT, var.name))
     return func
 
 

Still… all the other responses were kinda broken too. I e-mailed the pydap list and Roberto got back to me right away recommending that I update pupynere, the library that writes out the netcdfs. Success! I haven’t checked the other responses, but Roberto said that he’s working on going over them and making sure that the encoding is explicitly declared. Awesome!