James Hiebert's Blog

Setting up the PCIC Data Portal in a few easy commands

The data portal has been in drastic need of some refactoring and modularization. We want to be able to work on and test different components of the system independently, but this has not been possible with the original code base. It included several (many?) circular dependencies and was a bit of a monolith. After a couple weeks of work, I’m finally starting to tease apart a few of the pieces to be able to install and test the independently. It’s a lot of worth, but it’s showing it’s value almost immediately. Here’s an example of how to install part of the system, from scratch, in only three simple commands.

Command one: create a python virtual environment.

james@basalt ~/code/hg/pydap.handlers.pcic $ virtualenv --system-site-packages /tmp/testenv3
New python executable in /tmp/testenv3/bin/python2.7
Also creating executable in /tmp/testenv3/bin/python
Installing setuptools............done.
Installing pip...............done.

Command two: upgrade setuptools (to be able to use mercurial repositories natively).

james@basalt ~/code/hg/pydap.handlers.pcic $ /tmp/testenv3/bin/pip install -U setuptools
Downloading/unpacking setuptools from https://pypi.python.org/packages/source/s/setuptools/setuptools-0.9.8.tar.gz#md5=243076241781935f7fcad370195a4291
  Downloading setuptools-0.9.8.tar.gz (766kB): 766kB downloaded
  Running setup.py egg_info for package setuptools
    
Installing collected packages: setuptools
  Found existing installation: setuptools 0.6c11
    Uninstalling setuptools:
      Successfully uninstalled setuptools
  Running setup.py install for setuptools
    
    Installing easy_install script to /tmp/testenv3/bin
    Installing easy_install-2.7 script to /tmp/testenv3/bin
Successfully installed setuptools
Cleaning up...

Command three: install a package. In this actual case, I’m only installing a minor component which is, maybe, halfway up the dependency tree for the entire data portal application. But, the idea is that this will eventually (in a couple weeks) work for all of the required components.

james@basalt ~/code/hg/pydap.handlers.pcic $ /tmp/testenv3/bin/python ./setup.py develop
running develop
running egg_info
writing requirements to pydap.handlers.pcic.egg-info/requires.txt
writing pydap.handlers.pcic.egg-info/PKG-INFO
writing namespace_packages to pydap.handlers.pcic.egg-info/namespace_packages.txt
writing top-level names to pydap.handlers.pcic.egg-info/top_level.txt
writing dependency_links to pydap.handlers.pcic.egg-info/dependency_links.txt
writing entry points to pydap.handlers.pcic.egg-info/entry_points.txt
reading manifest file 'pydap.handlers.pcic.egg-info/SOURCES.txt'
writing manifest file 'pydap.handlers.pcic.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/testenv3/lib/python2.7/site-packages/pydap.handlers.pcic.egg-link (link to .)
Adding pydap.handlers.pcic 0.0.2 to easy-install.pth file

Installed /home/james/code/hg/pydap.handlers.pcic
Processing dependencies for pydap.handlers.pcic==0.0.2
Searching for pydap.responses.html==0.2dev
Best match: pydap.responses.html 0.2dev
Doing hg clone from ssh://medusa.pcic.uvic.ca//home/data/projects/comp_support/software/pydap.responses.html to /tmp/easy_install-fQqg5u/pydap.responses.html@d8689fab1694
Updating to d8689fab1694
Processing pydap.responses.html@d8689fab1694
Writing /tmp/easy_install-fQqg5u/pydap.responses.html@d8689fab1694/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-fQqg5u/pydap.responses.html@d8689fab1694/egg-dist-tmp-sY03yN
Adding pydap.responses.html 0.2dev to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/pydap.responses.html-0.2dev-py2.7.egg
Searching for pycds>=0.0.12
Reading https://pypi.python.org/simple/pycds/
Couldn't retrieve index page for 'pycds'
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Best match: pycds 0.0.12
Doing hg clone from ssh://medusa.pcic.uvic.ca//home/data/projects/comp_support/software/PyCDS to /tmp/easy_install-HY6_T7/PyCDS@0.0.12
Updating to 0.0.12
Processing PyCDS@0.0.12
Writing /tmp/easy_install-HY6_T7/PyCDS@0.0.12/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-HY6_T7/PyCDS@0.0.12/egg-dist-tmp-PSaUmW
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'
  warnings.warn(msg)
Adding PyCDS 0.0.12 to easy-install.pth file
Installing demo.py script to /tmp/testenv3/bin

Installed /tmp/testenv3/lib/python2.7/site-packages/PyCDS-0.0.12-py2.7.egg
Searching for pydap.handlers.sql==0.6dev
Reading https://pypi.python.org/simple/pydap.handlers.sql/
Best match: pydap.handlers.sql 0.6dev
Doing hg clone from ssh://medusa.pcic.uvic.ca//home/data/projects/comp_support/software/pydap.handlers.sql to /tmp/easy_install-FAjOnD/pydap.handlers.sql@9d5d7347ef67
Updating to 9d5d7347ef67
Processing pydap.handlers.sql@9d5d7347ef67
Writing /tmp/easy_install-FAjOnD/pydap.handlers.sql@9d5d7347ef67/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-FAjOnD/pydap.handlers.sql@9d5d7347ef67/egg-dist-tmp-iE06zC
Adding pydap.handlers.sql 0.6dev to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/pydap.handlers.sql-0.6dev-py2.7.egg
Searching for SQLAlchemy==0.8.3dev
Reading https://pypi.python.org/simple/SQLAlchemy/
Best match: SQLAlchemy 0.8.3dev
Downloading https://bitbucket.org/zzzeek/sqlalchemy/get/rel_0_8.tar.gz#egg=SQLAlchemy-0.8.3dev
Processing rel_0_8.tar.gz
Writing /tmp/easy_install-FleJy4/zzzeek-sqlalchemy-9fcb5348afcd/setup.cfg
Running zzzeek-sqlalchemy-9fcb5348afcd/setup.py -q bdist_egg --dist-dir /tmp/easy_install-FleJy4/zzzeek-sqlalchemy-9fcb5348afcd/egg-dist-tmp-eiNYls
warning: no files found matching '*.html' under directory 'doc'
warning: no files found matching '*.jpg' under directory 'doc'
no previously-included directories found matching 'doc/build/output'
zip_safe flag not set; analyzing archive contents...
Adding SQLAlchemy 0.8.3dev to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/SQLAlchemy-0.8.3dev-py2.7-linux-x86_64.egg
Searching for pydap.handlers.csv==0.3dev
Reading https://pypi.python.org/simple/pydap.handlers.csv/
Best match: pydap.handlers.csv 0.3dev
Doing hg clone from ssh://medusa.pcic.uvic.ca//home/data/projects/comp_support/software/pydap.handlers.csv to /tmp/easy_install-_rUcC1/pydap.handlers.csv@2ccf8be6114b
Updating to 2ccf8be6114b
Processing pydap.handlers.csv@2ccf8be6114b
Writing /tmp/easy_install-_rUcC1/pydap.handlers.csv@2ccf8be6114b/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-_rUcC1/pydap.handlers.csv@2ccf8be6114b/egg-dist-tmp-zVxeCF
Adding pydap.handlers.csv 0.3dev to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/pydap.handlers.csv-0.3dev-py2.7.egg
Searching for Pydap==3.2.1dev
Reading https://pypi.python.org/simple/Pydap/
Best match: Pydap 3.2.1dev
Doing hg clone from ssh://medusa.pcic.uvic.ca//home/data/projects/comp_support/software/Pydap-3.2 to /tmp/easy_install-KoZ4gC/Pydap-3.2@fc86c858629
Updating to fc86c858629
abort: unknown revision 'fc86c858629'!
Processing Pydap-3.2@fc86c858629
Writing /tmp/easy_install-KoZ4gC/Pydap-3.2@fc86c858629/setup.cfg
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-KoZ4gC/Pydap-3.2@fc86c858629/egg-dist-tmp-bl80JU
Adding Pydap 3.2.1dev to easy-install.pth file
Installing dods script to /tmp/testenv3/bin
Installing pydap script to /tmp/testenv3/bin

Installed /tmp/testenv3/lib/python2.7/site-packages/Pydap-3.2.1dev-py2.7.egg
Searching for PyYAML
Reading https://pypi.python.org/simple/PyYAML/
Best match: PyYAML 3.10
Downloading https://pypi.python.org/packages/source/P/PyYAML/PyYAML-3.10.zip#md5=b1a2b30cdf481da4249c917c3307f129
Processing PyYAML-3.10.zip
Writing /tmp/easy_install-KMW7vT/PyYAML-3.10/setup.cfg
Running PyYAML-3.10/setup.py -q bdist_egg --dist-dir /tmp/easy_install-KMW7vT/PyYAML-3.10/egg-dist-tmp-lgt0nw
In file included from ext/_yaml.c:223:
ext/_yaml.h:6:1: warning: "PyUnicode_FromString" redefined
In file included from /usr/include/python2.7/Python.h:85,
                 from ext/_yaml.c:4:
/usr/include/python2.7/unicodeobject.h:281:1: warning: this is the location of the previous definition
ext/_yaml.c: In function ‘__pyx_pf_5_yaml_get_version_string’:
ext/_yaml.c:1145: warning: assignment discards qualifiers from pointer target type
ext/_yaml.c: In function ‘__pyx_pf_5_yaml_7CParser___init__’:
ext/_yaml.c:2074: warning: passing argument 2 of ‘yaml_parser_set_input’ from incompatible pointer type
/usr/include/yaml.h:1367: note: expected ‘int (*)(void *, unsigned char *, size_t,  size_t *)’ but argument is of type ‘int (*)(void *, char *, int,  int *)’
ext/_yaml.c: In function ‘__pyx_pf_5_yaml_8CEmitter___init__’:
ext/_yaml.c:11065: warning: passing argument 2 of ‘yaml_emitter_set_output’ from incompatible pointer type
/usr/include/yaml.h:1829: note: expected ‘int (*)(void *, unsigned char *, size_t)’ but argument is of type ‘int (*)(void *, char *, int)’
zip_safe flag not set; analyzing archive contents...
Adding PyYAML 3.10 to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/PyYAML-3.10-py2.7-linux-x86_64.egg
Searching for simplejson
Reading https://pypi.python.org/simple/simplejson/
Best match: simplejson 3.3.0
Downloading https://pypi.python.org/packages/source/s/simplejson/simplejson-3.3.0.tar.gz#md5=0e29b393bceac8081fa4e93ff9f6a001
Processing simplejson-3.3.0.tar.gz
Writing /tmp/easy_install-3XSb7t/simplejson-3.3.0/setup.cfg
Running simplejson-3.3.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3XSb7t/simplejson-3.3.0/egg-dist-tmp-0AIi5c
zip_safe flag not set; analyzing archive contents...
simplejson.tests.__init__: module references __file__
Adding simplejson 3.3.0 to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/simplejson-3.3.0-py2.7-linux-x86_64.egg
Searching for requests
Reading https://pypi.python.org/simple/requests/
Best match: requests 1.2.3
Downloading https://pypi.python.org/packages/source/r/requests/requests-1.2.3.tar.gz#md5=adbd3f18445f7fe5e77f65c502e264fb
Processing requests-1.2.3.tar.gz
Writing /tmp/easy_install-sibNAa/requests-1.2.3/setup.cfg
Running requests-1.2.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-sibNAa/requests-1.2.3/egg-dist-tmp-HGgud2
Adding requests 1.2.3 to easy-install.pth file

Installed /tmp/testenv3/lib/python2.7/site-packages/requests-1.2.3-py2.7.egg
Searching for psycopg2==2.5.1
Best match: psycopg2 2.5.1
Adding psycopg2 2.5.1 to easy-install.pth file

Using /usr/lib64/python2.7/site-packages
Searching for numpy==1.7.1
Best match: numpy 1.7.1
numpy 1.7.1 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages
Searching for WebOb==1.2.3
Best match: WebOb 1.2.3
WebOb 1.2.3 is already the active version in easy-install.pth

Using /usr/lib64/python2.7/site-packages
Finished processing dependencies for pydap.handlers.pcic==0.0.2

An optional step 4: run all of the tests for this component. They pass!

james@basalt ~/code/hg/pydap.handlers.pcic $ /tmp/testenv3/bin/python ./setup.py test
running test
running egg_info
writing requirements to pydap.handlers.pcic.egg-info/requires.txt
writing pydap.handlers.pcic.egg-info/PKG-INFO
writing namespace_packages to pydap.handlers.pcic.egg-info/namespace_packages.txt
writing top-level names to pydap.handlers.pcic.egg-info/top_level.txt
writing dependency_links to pydap.handlers.pcic.egg-info/dependency_links.txt
writing entry points to pydap.handlers.pcic.egg-info/entry_points.txt
reading manifest file 'pydap.handlers.pcic.egg-info/SOURCES.txt'
writing manifest file 'pydap.handlers.pcic.egg-info/SOURCES.txt'
running build_ext
==================================================================== test session starts ====================================================================
platform linux2 -- Python 2.7.5 -- pytest-2.3.5 -- /tmp/testenv3/bin/python
collected 12 items 

tests/test_pydap_handlers_pcic.py:7: test_composite PASSED
tests/test_pydap_handlers_pcic.py:10: test_raw PASSED
tests/test_pydap_handlers_pcic.py:13: test_climo PASSED
tests/test_pydap_handlers_pcic.py:16: test_get_vars[input0-expected0] PASSED
tests/test_pydap_handlers_pcic.py:16: test_get_vars[input1-expected1] PASSED
tests/test_pydap_handlers_pcic.py:145: test_create_ini[EC_raw-1106200-expected0] PASSED
tests/test_pydap_handlers_pcic.py:145: test_create_ini[ARDA-109147-expected1] PASSED
tests/test_pydap_handlers_pcic.py:170: test_monkey PASSED
tests/test_pydap_handlers_pcic.py:181: test_404s[/EC/913/junk] PASSED
tests/test_pydap_handlers_pcic.py:181: test_404s[/EC/913.sql.html] PASSED
tests/test_pydap_handlers_pcic.py:192: test_returns_content PASSED
tests/test_pydap_handlers_pcic.py:213: test_returns_html_content PASSED

================================================================= 12 passed in 0.33 seconds =================================================================

The whole thing took 26 seconds from start to finish, which is pretty good.

blog comments powered by Disqus

Published

30 July 2013

Category

work

Tags