James Hiebert's Blog

PyCon: Day 3

Deliver Your Software in an Envelope

Behavioral Envelopes

Analagous to a flight envelope for aircraft. What is the behaviour as it relates to the speed and altitude.

A simple metaphore. Takes the full space and divides it into what is allowed and what is not.

Can we do this for software? "We think that we can."

Examples:

math.sqrt has a very well defined set of inputs

"System Requirements": e.g. RAM requirements for video game

e.g. HTTP servers: Lots of examples and what constitute valid requests/responses

If you send it random garbage, it will give you a forbidden input

e.g. Ether killer. Disallowed input.

Look at the system from both sides: input and output

output envelopes. Not talking about the output of deterministic processes, BUT random.random, has an envelope, but is not deterministic.

BlinkDB! Will give you less accurate results based on how quickly you want them.

Games or financial systems will often have unspecified behaviour because you don't want cheaters to game the system.

Weak theseis: You should state your system's beahvioural envelope.

String thesis: Your statement of your system's behavioural envelope should be your only statement of your system's behaviour.

E.g. Java type system. Haskell type system. Haskell version of map, has type annotations that go along with it. Both a type checker and new users can reason about it.

Tests help this. They're kind of discrete fence posts along the continuous envelope. They're useful and they sort of show the line of the envelope. Documentation allows you to fill in the gaps.

E.g. parents giving boundaries to the children, so that they could keep an eye on the kids. Curious, because it defines the continuous lines based on a few discrete points. Test conditions specify the points, while types define continous curves through the space.

When one is supposed to write tests near the boundaries, that is because your tests are almost defining the boundaries of the envelope.

Legal description that this software cannot be assumed to do anything. Usefully, we want people to believe that our software is good for something. Demos are great for that. Tests are demos. Why not ship the tests with your code, so that people can read machine-checked code that verifies what your system does.

  • Types, tests, documentation, define that envelope.
  • Less (Guarantees) is More (Freedom to Keep Those Guarantees).
  • Fewer promises makes everyone else happier. Because it reserves future growth. If your envelop starts small, you have room to grow. If it starts large, you have lots of promises to keep forever more.

E.g. httplib

"HTTPResponse class does not enforce this state machine, which implies sophisticated clients may accelerate the request/response pipeline."

This means that people will depend on behaviour that isn't actually promised. That sucks.

Java's Object methods:

Object.clone
Object.finalize # like __del__ in python, but there's no guarantee that this will ever run

Good Examples

HTTP (the protocol)

There were originally 15 response codes. An enumerated finite set. But those values were spread across a range of 1000, to give separate classes. Leaves room for future growth. Reserved what you should do when you get a response that you don't understand.

concurrent.futures

Can fire off a bunch of requests in parallel and then go do stuff while they're working

  • ThreadPoolExecutor
  • ProcessPoolExecutor

my_dict[my_key] = new_value

  • Is this thread safe or not? It depends on the implementation details.
  • Source code is not the best documentation. The behavioural envelope is.

Summary

  • Provide small, well-defined envelopes.
  • Only rely on the stated envelop that others provide.

Questions

  • Q: Disallow tests that ask for "correctness". Because every time you expand your system, you expand that envelop as to what is "correct".
  • Q: What tools do you use to do this now? A: I work on version control software (e.g. mercurial). hg's test suite is "curious"... looks like a large shell script. A2: pylint which is close to static analysis, unittest... I have lint check my documentation. Because in old code that I've taken over and had to maintain, the inconsistencies and problems in the behavioural envelope sneak by when the docstrings are missing. I'm my own best friend and not critical of myself, so if I can outsource that criticism, that will work better.
  • Q: What's your experience with taking over software. A: First thing someone did was
  • Q: Use assert envelopes? A: I use them all of the time. But raising an exception is often the better tool. asserts can be turned off so they scare me.
  • Q: I write C++ code with doxygen, and there is stuff for having preconditions and postconditions. Does sphinx use that? A: I've never seen that as a feature. A2: Perhaps, lack of formals system for that has pressured me to ensure that the type system itself satisfies those constraints. Therefore I favor the use of objects so that by the time the server gets an object, you have some assurance that the object has a type.
  • Q: contextlib can force pre and post conditions. I haven't used it a lot, but it is an elegant way to express invariants around a function and another block of code. Just use a with statement.
  • Q: numpy had this problem and they solve it (somehow). Numpy doc pacakge automatically converts the documentation into SWIG statements.

PostgreSQL is Web Scale. Really :)

Background

  • Python user since version 1
  • Postgres user since before it was acquired SQL in its name to become PostgreSQL
  • first DBA and Datbase Architect at Skype
  • full0time PostgreSQL consultant since 2007 as port of 2ndQuadrant
  • Tech. Advisor to Skype VP

Why NoSQL

  • Simplicity to implement (originally)
  • "Let's use files for speed"
  • scalability at storage layer

What NoSQL usually sacrifices

  • ACID

using postgresql as a key/JSON-value store

This presentation was bad and seemed very basic. I left.

Hickhiker's Guide to Participating in Open Source

...

Advice to coders

  • Can't just waltz up to a project and make major changes.
  • (Stable/Working) > (your Ego)
  • Expect to get Rejection but playful, friendly...
  • Learn to take "feedback" (i.e. cirticism)
  • Be Overly Modest
  • "My patch is amazing"
  • You are Here by Choice. Engage Socially.
  • Duration of participation counts.
  • Ask your employer to give back to your project.

Advice to Non-coders?

  • FOSS needs non-Coders. Should be 50% - 50% (e.g. doctors w/o borders... 50% are not doctors)
  • Code Review (learning). Every project needs designers, content makers, writers, translations.
  • Be modest and be OK with yourself. Be confident you can do something they can't. Beware of gaping holes in their knowledge outside code and be gentle about this.
  • Learn to use Github.

Other

  • How to be a Project Owner. I don't know. Ask Carl.
  • Installation and dependencies. Ensure it's possible to Run/Install!!!
  • First time?: Don't expect contributors. "Adverising" helps. Maintenance and care helps. Enough code to convince...
  • Getting other people to help...
  • Give recognition!
  • Don't automatically Reject/Revert. Don't let people find out through an automated system. Drop a quick note, there's usually a reason.
  • Always be cordial; you never know who's watching.
  • Embrace your churn. Average involvement is 3 months to 2 years.
  • Burn out; take breaks.
  • Audience participation! Someone come up right now!
  • Communicate! Communicate! 95% of problems are solved by communication!
  • Ask/Explain: Be terse, explain and get to the point.
  • Set expectations. Do what you say that you're going to do.
  • Don't agonize... save yourself the pain.
  • Say, "I don't know."
  • Ask: if you're stuck for 30 minutes, go to :IRC, mailing lists, [one other]
  • Don't criticize people's life decisions... you're on each other's team. Don't criticize technical fixes/solutions that are currently working.
  • Don't criticize other FOSS projects without contacting the project first. Send an e-mail or submit a ticket.
  • Don't get disheartened. All mistakes will eventually be washed clean by time and entropy.
  • Say thanks. Send an e-mail, tweet, say thanks to peopel for all their hard work.

Conclusion

Thank you to the "Python Diversity Grant"!

  • Q: People were jerks when I tried to contribute. But... I don't know what I do well for the FOSS community. I don't know how to translate my job responsibilities to the FOSS community. A: Find communities where they aren't jerks. At the sprints, they want you to be there.
  • Q: I'm terrified and am just getting into it. It's easy to do typos and fixes, but how do I make the next step? A: Hang around. You'll find that your understanding just comes from working on it and understanding it better.
  • Q: Github projects are easy to contribut to. What about projects that aren't? A: No. Version control is hard and we're still working on a problem. E.g. Linux submits all of its patches by e-mail.
  • Q: Your passion is admirable. OSS projects are well-honed projects. Where do I concern myself. A: That's a misconception. There are only a small number of projects that are very serious and don't want outside help.

Set your code free: releasing and maintaining an oepn-source project

Carl!

Assumptions

  1. You have code!
  2. You want me to use it! via pip install (there are other ways, outside of the scope of this talk)
  3. You want me to contribute to it.

Look up some documentation on the Internet. Goooooooood luck with that. There will be various advice and documentation. When you're doing something for the first time, too many options are bad for motivation.

0 - Awesome in 20 minutes or less.

A good project will have:

  • HTML docs
  • Responsive theme.
  • Tests that run when you push to your repo.
  • A project that is welcoming to others.

All the things

  1. Structure
  2. Licenes
  3. hosting.
  4. docs.
  5. testing and CI
  6. packaging

  7. community

    PyFly docs/ pyfly/ init.py tests/ LINCENSE.txt MANIFEST.txt README.rst setup.py

License

  • Conditions.
  • Defaults is all rights reserved. Which means that I can't use it.
  • If you aren't sure, use BSD or MIT (according to Carl).
  • Or (L)GLP, Apache, MPL.
  • Don't use anything else. The goal of a license is to clarify concerns. If you use something else, you're giving others a reason to question whether they should use your code.

This should go in the LICENSE.txt file.

Hosting

GitHub, because that's where the people are. Just do it.

Docs

If it's not documented, it doesn't exist. I like reading code. But if reading your code is the first step. I'll probably write it myself.

  • Use Sphinx
  • Host at readthedocs.

readthedocs

Give readthedocs our info Go to GitHub and enable readthedocs. It gives you lots of awesomeness.

Testing

  • If it's not tested, it's broken.
  • Dependency matrix is large, especially if we want to support multiple versions of python versions plus our version. Better actually build the virtualenvs and do it.
  • Use tox. With one command it will create a bunch of venvs and run your tests.

tox.ini

[tox]
envlist = py27,py34,pypy

[testenv]
deps = pytest
commands = py.test

tox will decide whether it passes for fails, just based on the return code.

When we run tox, it will create a package of our project, installs our deps, installs our package, runs the tests, and then gives the output of what passes/fails. It gives a smiley face!

Pull request? Use travis-ci.org for continuous integration. It will provide continuous interation for free with public github repos. Tell travis how to run our test with a .travis.yml file. Similar to tox. There are tools to convert tox.ini to .travis.yml. Carl just maintains them both in parallel. They're simple. It's not hard to do.

Also, every time someone sends you a pull request, travis will run the tests and github will tell you whether it passes or fails.

Packaging

setuptools is technically 3rd party, but in reality everyone uses it, so it will be there.

MANIFEST.in

setup.py will include all python files, but everything else must be included in MANIFEST.in.

tag before release

git tag v0.1

push tags to github

python setup.py sdist

New wheel format. So if you have compiled extensions, it makes it easier for people to install (they don't have to build it).

Once you have your sdist, you can install directly to pip

pip install dist/PyFly-0.1.2.tar.gz

Pip can install to PyPi, but it uses non-encrypted channel. Use twine. pip install twine twine upload dist/PyFly-0.1.2.tar.gz pip install PyFly

Win!

http://packaging.python.org IRC: #pypa freenode.org

Community

Find ways to make your contributers and yoru live easier. Value people's time.

Communicate clearly changes that will affect their lives. Semantic versioning.

X.Y.Z

X for breaking changes. Y for backwards-compatible feature additions. Z for bugfixes.

Keep a changelog WHen they see backward incompatible changelog, they can look for something interesting

CHANGES.rst

Cannot auto-generate this because you need to interpret how it's important to them.

CONTRIBUTING.rst

HOw to get set up for dev How to run the tests What to include in a bug report. Coding Standards, test coverage standards... Github will find it and include it.

Keep the tests passing

I want to work from a clean slate. If your tests fail, my motivation for contributing is gone.

Give quick feedback. If you get a bug report or feature request, you may not have time, but you should at least give a one line request, explaining what the situation is.

Give credit. When people pitch in, you should give them props. Commit messages, authors files, tweets, etc. If you've gotten good commits from people, give them commit access. Someday you're going to be tired of maintaining this. Preparing someone to take over the project when you're done with it.

Be nice. Even if it's irritating or doesn't work or whatever, it still represents a contribution and someone who took the time, and wants your project to be better. For every one person you talk to in a public way, there may be 10-20-100 other people that may see it and decide whether they should interact with you in the future.

http://oddbird.net/set-your-code-free-preso

Questions

Q: git tag. Don't just use git tag, use the git message. Also use -s and sign your tags and sign your PyPi uploads. Also make sure that copyright is bundled with license. Be careful about whether deps have compatible license. Semantic versioning, that's awesome. Q: How do you feel about using github as a way to get started on getting contributors. I.e. you need assistance, but you have some boilerplate. A: Usually you get contributors because you've built something useful. But probably an idea or whatever probably won't gain traction. Q: How to maintain MAINIFEST.in automatcially? I've broken libraries. A: Yeah, it's a problem. Good reason to check your sdist. If you have a break in the release, bump the version and move on. Once it's out, it's out. Q: Will this be automated in the future? A: It's all more automated than it was in the last 2-3 years. I hope that things will get easier.

-----
blog comments powered by Disqus

Published

28 April 2014

Category

work

Tags