Wednesday, January 20, 2016

The Clean Architecture in Python

Even design-conscious programmers find large applications difficult to maintain. Come learn about how the recently propounded “Clean Architecture” applies in Python, and how this high-level design pattern fits particularly well with the features of the Python language and answers questions that experienced programmers have been asking.
The Clean Architecture in Python, Brandon Rhodes, PyOhio 2014

Tuesday, December 1, 2015

PyMongo query logging

I needed to see what queries are sent to the server in my MongoEngine application.

I made one logger here: https://github.com/warvariuc/python-mongo-logger based on https://gist.github.com/kesor/1589672

Recently I improved the solution, but did not commit to the repo, because I put this code directly into the project instead of having git+git://github.com/warvariuc/python-mongo-logger
 in requirements.pip file:

from __future__ import absolute_import, print_function, unicode_literals, division

import logging
import time
import traceback
import inspect

from pymongo.mongo_client import MongoClient
from bson import json_util


logger = logging.getLogger('mongologger')


def activate(until_modules=('pymongo', 'mongoengine'), stack_size=3):
    """Activate Mongo-Logger.
    Args:
        until_modules (list): list of top level module names until which the stack should be shown;
          pass an empty sequence to show the whole stack
        stack_size (int): how many frames before any of `modules` was entered to show; pass
          -1 to show the whole stack or 0 to show no stack
    """
    # monkey-patch methods to record messages
    MongoClient._send_message_with_response = _instrument(
        MongoClient._send_message_with_response, until_modules, stack_size)
    return logger


def _instrument(original_method, until_modules, stack_size):
    """Monkey-patch the given pymongo function which sends queries to MongoDB.
    """
    def _send_message_with_response(client, operation, read_preference=None,
                                    exhaust=False, address=None):
        start_time = time.time()
        result = original_method(client, operation, read_preference=None,
                                    exhaust=False, address=None)
        duration = time.time() - start_time
        try:
            stack = ('\n' + ''.join(get_stack(until_modules, stack_size))).rstrip()
            logger.info('%.3f %s %s %s%s', duration, operation.name, operation.ns,
                        json_util.dumps(operation.spec, ensure_ascii=False), stack)
        except Exception as exc:
            logger.info('%.3f *** Failed to log the query *** %s', duration, exc)
        return result

    return _send_message_with_response


def get_stack(until_modules, stack_size):
    """
    """
    if not stack_size:
        return []
    frames = inspect.stack()[2:]
    frame_index = None
    for i, (frame, _, _, _, _, _) in enumerate(frames):
        module_name, _, _ = frame.f_globals['__name__'].partition('.')
        if module_name in until_modules:
            frame_index = i
        elif frame_index is not None:
            # found first frame before the needed module frame was entered
            break

    if frame_index is not None:
        del frames[:frame_index + 1]
        if stack_size >= 0:
            del frames[stack_size:]

    stack = [(filename, lineno, name, lines[0])
             for frame, filename, lineno, name, lines, _ in frames]

    return traceback.format_list(stack)

You activate the logger in the logging configuration like this:

import mongologger
LOGGING['loggers'][mongologger.activate(stack_size=0).name] = {
    'level': 'INFO',
    'handlers': ['stdout'],
    'propagate': False,
}


Saturday, November 28, 2015

Export environment variables for post-build steps in Jenkins

I needed a way to pass environment variables to post-build steps in Jenkins. HipChat notification plugin allows to use environment variables in its templates, but I cannot use commands there, so the template flexibility is quite limited.

So I wanted to make the message text in a build step and then pass it to the post-build HipChat notifications step.

There is EnvInject Plugin to accomplish this, but I found it to be not user-friendly -- I had to use two steps: 1) echo values to a properties file; 2) then use EnvInject to read the values from the file and export them.

I found a solution that suits me better. In build steps I only create/update the properties file:

echo "hipchat_message=Server build succeded: <a href='https://$SERVER_NAME/'>$SERVER_NAME</a> (<a href='$BUILD_URL'>Job</a>)" > "$WORKSPACE/postbuild.props"

and then in the first post-build step use this Groovy script:

/*
Inject environment variables using Groovy because EnvInject plugin is not user-friendly
*/

import hudson.model.*

def console = manager.listener.logger.&println

// read the props file
def props = new Properties()
new File("${manager.envVars['WORKSPACE']}/postbuild.props").withInputStream { 
    stream -> props.load(stream) 
}

props.each{
    key, value -> console("${key}:${value}")
    def pa = new ParametersAction([
        new StringParameterValue(key, value)
    ])
    manager.build.addAction(pa)
} 

Friday, November 20, 2015

Cache and SSL in Chrome on Ubuntu Linux

I had a hard time testing if cache headers I've added to static files are working. Firefox was working as supposed, while Chrome wasn't.

Here is the Nginx configuration I was testing:

    # static files
    location ~ ^/(***)?$ {
        root ***;
        try_files $uri /;
        add_header Cache-Control "max-age=3600";
        gzip on;
        #etag on;
        access_log off;
    }

I was trying different combinations to find out why sometimes the files where taken from cache (looking to developer's console in Chrome) and why other times Chrome was always making requests with response 200.

My tests showed that my Chrome was using cache if both Cache-Control and ETag headers where present. The problem is that if `etag on` directive is enabled together with `gzip on`, etags are not generated: https://trac.nginx.org/nginx/ticket/377

When I disable gzip and enable etag, cache works ok.

But my colleagues say that in their browsers, including Chrome on OSX cache works as expected.

It turned out that the server I was testing on had a self signed certificate, and Chrome has issues with caching in such cases: https://code.google.com/p/chromium/issues/detail?id=110649

I tried to export the certificate and import it back as a trusted certificate, but it didn't work. They say:

Google Chrome in Linux doesn’t have a SSL certificate manager, it relies on the NSS Shared DB.


Now cache works as expected in my Chrome on Ubuntu Linux.

Thursday, November 5, 2015

Run SketchUp Make 2015 in Ubuntu 15.10

1. Install wine
2. Download SketchUp Make 2015 Windows 32 Bit from http://www.sketchup.com/download/all
3. In a terminal
export WINEARCH=win32 WINEPREFIX=$HOME/sketchup
# select Windows 7
winecfg
# run installation
wine SketchUpMake-en.exe
# run SketchUp
wine C:/Program\ Files/SketchUp/SketchUp\ 2015/SketchUp.exe /DisableRubyAPI
Menu crashes the program, so press Ctrl-O to open a file.

I've installed Wine 1.7.44 and the program doesn't crash when selecting File menu.

Wednesday, May 20, 2015

Nina Zakharenko - Technical Debt - The code monster in everyone's closet - PyCon 2015

Technical debt is the code monster hiding in everyone's closet. If you ignore it, it will terrorize you at night. To banish it and re-gain your productivity, you'll need to face it head on.

Alex Gaynor - Techniques for Debugging Hard Problems - PyCon 2015

Sometimes your programs have bugs. Often they're shallow things, simple AttributeErrors or TypeErrors. Sometimes they're large, complex, and nearly impossible to debug. This talk explores techniques for figuring these out.