Nesting: Setting up your Python Environment

When you begin using Python a lot, you realize you’re creating (or re-creating!) the same functions over and over again.  I don’t know how many times, for example, I have re-Googled and re-typed the recipe from the magnificent and eccentric Python library BeautifulSoup to get text out of an HTML document, as when I want to create a simple search index.

I also end up needing to find this meta-recipe for loading my own frequently-used functions and helpers into my Python environment. So I thought I’d write up, in one place, how to set up your Python environment so that your own libraries and code snippets are in scope, and ready for you to use. It’s simple and I really wanted to get it down.

Feathering the Python nest

What we want to end up with is a Python environment in which tools and script we have already written are already loaded and at our fingertips. Start Python—or iPython, or some other Python-based environment—and your stuff is there.

In my environment I have created a module called brownhen in which functions I reuse are defined. You can call yours  anything you want—bobby, greatstuff, powertools, whatever. But it’s important to “name space” these functions and utilities to remind yourself where these things come from and what package they’re a part of. In this tutorial, my stuff is in a module called “brownhen”.

So this recipe describes how to:

  1. Organize your useful stuff into modules—a very good idea in any case
  2. Set up your environment to find your modules when Python starts (PYTHONPATH)
  3. Tell your environment not only where your modules are, but to pre-load them when you fire Python up (PYTHONSTART)

When I start up the iPython interpreter, which I keep up most all the time and sort of live in, my environment welcomes me and lets me know that my imports are working:

When Python tells me that it has 2) found and 3) loaded utilities I have 1) organized into my own module(s), it means that I can easily reuse my own.

PYTHONPATH

Where does Python look for the libraries and code it needs when it starts up? Where can it look for new stuff when you use import statements? It doesn’t just look everywhere on your laptop. That’s bad form and takes too long. 

The answer is that Python reads the system’s PYTHONPATH environment variable for any paths that should be added to what it already knows about, which typically gets defined when you install Python for the first ime. I am using the Anaconda distribution of Python right now, so the scope of Python’s searching is basically contained to the directory where Anaconda put down a series of executables, libraries, and tools:

Where does Python look for the libraries and code it needs when it starts up? Where can it look for new stuff when you use import statements? It doesn’t just look everywhere on your laptop. That’s bad form and takes too long.

The answer is that Python reads the system’s PYTHONPATH environment variable for any paths that should be added to what it already knows about, which typically gets defined when you install Python for the first ime. I am using the Anaconda distribution of Python right now, so the scope of Python’s searching is basically contained to the directory where Anaconda put down a series of executables, libraries, and tools:

To add to the list of places that Python searches when you start up, so that import statements can work without a hitch, edit the system PYTHONPATH variable and tell Python where else it should look:

Add the parent directory of “brownhen”, or “bobby”, to PYTHONPATH by editing your .bash_profie file on Mac, your .bashrc file on Linux, or your system environment variables on Windows:

If you put your stuff in a directory like ~/Dropbox/Programming/packages/brownhen/, then put the following (on Mac) into the file ~/.bash_profile:

This tells Python to look there when it starts up.  Already, you’ve set things up so that you can reach your brownhen stuff with import statements:

PYTHONSTART

Go one step further by not only telling Python where your stuff is, but asking Python to load it when it starts up. Use the PYTHONSTART environment variable for this:

This tells Python that there’s a particular script that should be executed when Python starts up. In this case, the script has one line, and that is the import statement that pulls the brownhen fabulousness in and makes it available. This is pystart.py in its entirety:

This tells Python to load the brownhen module, which can define functions and execute things right off the bat if you want. For this tutorial, my brownhen module file (see below) looks like this:

See that this file:

  • Imports a library—BeautifulSoup
  • Prints a statement to output to let you know it’s been loaded 
  • Defines two functions, one a test (hello) and the other we want to make sure and have available in all our Python sessions (textify)

With things set up in this way, I can start Python and begin to use my textify() function right away:

It works! I’ve got my environment set up so that frequently-used functions are 1) organized into a module, which 2) Python can find, and which is 3) loaded when I start up. Power. 

But how does this work? What’s a module again?

Python modules and module loading

The Python docs on this are good, and there’s lots of online support about modules and packages in Python, but just a word about it to round things out here:

Any directory in your path—in the list of directories you tell Python it should search via PYTHONPATH—is understood to be not just a directory but a Python module when it has a Python file in it called __init__.py (two underscores on either side of “init”):

This file __init__.py registers the directory “brownhen” as a module in Python, but it also gets executed as that module is loaded. So if in your PYTHONSTART file you import brownhen, as I have, you execute this file. 

In your module file, you can set define subdirectories as packages and do lots of other things, but know that this module initialization file gets execute by Python when the module is loaded, and is where you can put your favorite functions and tools. 

Syncing books, rising texts

Getting back to the text with ebooks

There must be a name in business school for the curve of ebook reading v. real books, for the shape that flares up at the beginning for individual readers, drops way down, possibly to zero, and then gradually climbs back a bit to some lower, steady level, like this:

Percentage of reading on device over time

Every ereader I know experiences some version of this curve. If we do it at all, we fall in love with the novelty of ereading, the cost and time savings, the ease of carrying around a pile of books on one device. The features of the ereader itself, like its bed-time back-lighting. But all or most of that leeches away in the face of this creeping feeling that we’re not actually reading, not actually absorbing the words as we flip through them, not seeing scenes, or having the recommended catharses, not getting into the text.

I’m on the steady, later part of the curve now, reading maybe half of my books and some magazines on a Kindle 1 device. The whiz-bang novelty has worn off, and so I feel like I’m finally in a position to appreciate the genuinely useful features of a digital reading device, the ones that are not pale versions of the book but advantages:

  • Searching in an ebook is real. It’s fantastic. The more you do it, the more you use this habit you acquired from web-surfing to locate, say, the first reference to a character, the name of a town.
  • e-Ink is really nice now. I tried reading on an Android tablet and the reflective glare and dim screen made that a non-starter outdoors.
  • Back-lighting and font control: I’m getting older! It’s nice to read on a device whose illumination, font-size and layout I can control. Keeps me from having to use readers. Plus I can read at night and disturb my wife [less].
  • Goodreads.com integration: If you can get over how bummed you are about how big and monopolistic Amazon is with the experience of reading, you can like how your lists on Goodreads.com–to-read, currently reading, etc.–are integrated into the ereader. A related item is that you can share your highlights and notes in the ebook as its own text, and can see sections and passages in a book that have been highlighted a lot.
  • Send content to your device to read later.

The feature of Kindles that I was most eager to try is this WhisperSync, which is a bridge between an ebook and its audio version. Amazon, having purchased Audible, allows you to move back and forth between the ebook and audiobook versions seamlessly, reading then listening, then reading. (The audio version is often advertised for this at a cheaper, “buy also” cost.)

For me, the whole point of returning to ebooks after having fallen down the ereading curve above, like everybody else, was not to take the place of actual books but to get a digital reading experience that wasn’t smothered in sidebar ads, cloying, embedded “read also” links, and bad layouts; hidden behind paywalls or hopelessly short or slapdash. I want digital long-form. I send myself readable versions of articles and focus on them as texts.

The ereader is pretty good for this. As you get used to it, it keeps improving, in my opinion.

And when you sync an ebook across the Kindle, the Audible version on your headphones 2, and maybe also the Kindle app on your phone for a quick few pages at the grocery store, it gets….really different, and maybe great! You can read a chapter, then hear the next one read by the author in a voice that’s completely consistent with your own internal one. It gets to be (again) like: I’m going to dive into the book. Too early to tell yet, but maybe syncing between these different formats can be even more absorbing, can reclaim “the text” for what it was before all these devices and distractions started diverting us.

Notes:

  1. Which reading platform to use—the Kindle, the Nook, or another device, like the Kobo—is another story, and one I want to write about soon, since I used to do pro-indy, anti-Kindle work for bookstores, have owned various Nooks as well, and now use this Kindle.
  2. You can use Bluetooth or aux headphones in newer Kindles

Website hacks and craft Web development

For the last several weeks, a few of the websites I maintain, including this one, have been getting absolutely buffeted by exploits and malware attacks. These attacks drop redirects in the headers and footers, throw up those fake blue “ATTENTION Microsoft” windows that take over your browser, and in general wreck the sites and make them do what the hackers want, whatever that is.

I change all the usernames and database logins, delete pernicious PHP files, chmod the wide-open directories. It keeps happening. It’s gotten so I think it’s not (just) a negligent webmaster like me, but something exploitable in my ISP, some way they’re not clamping down, some way-hidden hole I haven’t found yet. Because even when I change everything these exploits keep re-spreading. So it’s bad. It’s frustrating and it makes me throw up my hands about manual website maintenance. And about writing things on the web, since brownhen.com is like my (very occasional) journal.

But then I think: the good side of this is all this checking and fixing, this SSHing and tail-ing and chmod-ing. Like a fisherman darning his nets, I have to go in every day or so and look around, look at the logs, add IPs to the .htaccess (which is silly because hackers grab hundreds/thousands of computers, like mine, to drive their exploits, so it’s not like you’re reaching the guy’s laptop or anything), clean out the now-familiar fake files ( /wp-admin/user/exdbpabq.php is not a valid file from WordPress, for example). It’s like weeding a yard, sharpening your tools.

And it’s zen-like and pleasant like that. And direct. And craftsman-like. When I get over my frustration, I really like this mending and pruning and sharpening. What doesn’t get old is the directness of a web server and a shell, your favorite Unix editor, the activity on the site itself legible in logs, the cat-ing and bashing 1. Craft website development.

Notes:

  1. I found this cool Bash script that watches when new files are written to your website directory. I adjusted and am watching the intermittent “Waiting for changes” notes scroll down the terminal. All clear for now? Sorry this site has been down or abusing you when you visit.

    #! /usr/bin/env bash
    
    FILELIST=/tmp/filelist
    MONITOR_DIR=/home/usr/local
    
    [[ -f ${FILELIST} ]] || ls ${MONITOR_DIR} > ${FILELIST}
    
    while : ; do
        cur_files=$(ls ${MONITOR_DIR})
        diff <(cat ${FILELIST}) <(echo $cur_files) || \
             { echo "Alert: ${MONITOR_DIR} changed" ;
               # Overwrite file list with the new one.
               echo $cur_files > ${FILELIST} ;
             }
    
        echo "Waiting for changes."
        sleep $(expr 60 \* 2)
    done