Friday, June 24, 2016

Scott Sanderson, Joe Jevnik - Playing with Python Bytecode - PyCon 2016

Ever wondered what Python is actually doing when it executes your code? Want to learn to hand-craft artisanal Python bytecode? In this talk, we explain CPython's internal code representation, and we demonstrate techniques for modifying code objects for fun and profit.

Thursday, June 23, 2016

Matt Bachmann - Better Testing With Less Code: Property Based Testing With Python - PyCon 2016

Standard unit tests have developers test specific inputs and outputs. This works, but often what breaks code are the cases we did not think about. Property based testing has developers define properties of output and has the computer explore the possible inputs to verify these properties. This talk will introduce property based testing and provide real world examples and patterns.

Wednesday, June 15, 2016

Brett Slatkin - Refactoring Python: Why and how to restructure your code - PyCon 2016

As programs gain complexity, it becomes harder to add features and fix bugs. Reorganizing code is an effective way to make programs more manageable. This talk will show you Pythonic ways to do the most imporant ""refactorings"": Extract variables with __nonzero__; Change signatures with *args and **kwargs; Extract fields and classes with @property; Create stateful closures with __call__; and more!

Wednesday, June 1, 2016

Ned Batchelder - Machete-mode debugging: Hacking your way out of a tight spot - PyCon 2016

When chasing mysterious bugs, it's helpful to use all the tools at your disposal. We'll explore ways to use Python's dynamic tools to help track down the cause of head-scratching problems in large systems. Tools include the inspect module, monkey-patching, trace functions, and the Python mechanisms at work behind them all.

Tuesday, May 3, 2016

MongoDB dump/restore without temporary files

In MongoDB 3.2 new option --archive is available in mongodump and mongorestore which allow to to copy databases without temporary directories:

mongodump --host=localhost:37017 --db=userstate_10b --archive | mongorestore --host=localhost:27017 --drop --archive
MongoDB 3.2 also has a new --gzip option which in this case doesn't make sense to use, because mongodump and mongorestore are running on the same machine.

Saturday, April 9, 2016

Fixing HTTP latencies for `.local` hosts in /etc/hosts in Mac OSX 10.11.4 (El Capitan)

I have a problem on my Mac.

So I have a web-server on my localhost under `app.local` virtual server in Nginx config. When I make requests to it I notice that it takes cca. 5 seconds per requests. Looks pretty much as a timeout for DNS resolution.

The problem is known and there are several solution in the Internet. But only this helped me:

`etc/hosts`:

127.0.0.1 app.local
::1 app.local

So this trick is to specify IPv6 address too. And put hosts one per line -- hosts after the first one are ignored (a bug?).

Tuesday, March 22, 2016

Environment variables injecter for Jenkins

Based on my previous post about injecting environment variables in Jenkins to be used between steps and in post-build steps.

I created a groovy script which injects passed environment variables and even treats them as shell comands.

inject_env_vars.groovy
// Inject environment variables using Groovy

import hudson.model.*
import hudson.AbortException
import groovy.json.StringEscapeUtils


def build = Thread.currentThread().executable


def injectEnvVars(envVars) {
    for (item in envVars) {
        build.addAction(
            new ParametersAction([
                new StringParameterValue(item.key, item.value)
            ])
        )
    }
}

def bash(cmd, env) {

    cmd = cmd as String

    // create a process for the shell
    pb = new ProcessBuilder(["bash", "-c", cmd])
    // make job workspace directory as the current one
    pb.directory(new File(env['WORKSPACE']))
    pb.environment().putAll(env)
    // capture messages sent to stderr
    pb.redirectErrorStream(true)
    shell = pb.start()
    shell.getOutputStream().close()
    // capture the output from the command
    def shellIn = shell.getInputStream()

    def reader = new BufferedReader(new InputStreamReader(shellIn))
    def builder = new StringBuilder()
    while ( (line = reader.readLine()) != null ) {
       builder.append(line)
       builder.append(System.getProperty("line.separator"))
    }
    result = builder.toString()

    // wait for the shell to finish and get the return code
    def exitStatus = shell.waitFor()

    try {
        shellIn.close();
    } catch (IOException ignoreMe) {}

    if (exitStatus) {
        throw new AbortException(result)
    }
    return result
}

def readPropsFile(filePath) {
    // https://en.wikipedia.org/wiki/.properties
    def props = new Properties()
    new File(filePath).withInputStream {
        stream -> props.load(stream)
    }
    return props
}

def readEnvSh(filePath) {
    vars = [:]
    file = new File(filePath)
    file.eachLine { line ->
      def matcher = (line =~ '^(.*)=(.*)$')
      if (matcher) {
        def name = matcher.group(1)
        def value = matcher.group(2)
        if (value.startsWith('"')) { value = value[1..-2] }
        vars[name] = value
      }
    }
    return vars
}

// add some useful environment variables
env = build.getEnvironment(listener)
if (!env.containsKey('BUILD_USER')) {
    def userCause = build.getCause(hudson.model.Cause$UserIdCause)
    def userName = userCause?.userId ?: 'Jenkins'
    injectEnvVars([
        'BUILD_USER': userName,
        'JOB_DIR': "${env.WORKSPACE}/../../jobs/${env.JOB_NAME}",
        'BUILD_DIR': "${env.WORKSPACE}/../../jobs/${env.JOB_NAME}/builds/${env.BUILD_ID}",
    ])
}

// inject build result string: http://javadoc.jenkins-ci.org/hudson/model/Result.html
injectEnvVars(['BUILD_RESULT': "${build.result}"])

for (item in binding.variables.clone()) {
    def varName = item.key
    def varValue = item.value
    if (!(varValue instanceof String)) {
        // skip values injected by Jenkins Groovy plugin
        continue
    }
    if (varName == '_') {
        // run the given script code
        new GroovyShell(
            new Binding([
                'env': build.getEnvironment(listener),
                 'injectEnvVars': this.&injectEnvVars,
                 'readEnvSh': this.&readEnvSh,
            ])
        ).evaluate(varValue)
    } else {
        varValue = StringEscapeUtils.escapeJava(varValue)
        varValue = bash("echo \"${varValue}\"", build.getEnvironment(listener))
        injectEnvVars(["${varName}": varValue])
    }
}

Usage examples:

hipchat_message='${BUILD_USER} <a href="$BUILD_URL">started deploying backend</a> to <b>${project} ${target}</b>'
evaluate(new File("/home/jenkins/workspace/devops/inject_env_vars.groovy"))

hipchat_message='$(/home/jenkins/workspace/devops/deploy/get_deploy_info.py)'
evaluate(new File("/home/jenkins/workspace/devops/inject_env_vars.groovy"))


// variable `_` is considered to contain script code inside `inject_env_vars.groovy`
_='''
_env = readEnvSh("${env.WORKSPACE}/env.sh")

if (env['BUILD_RESULT'] == 'SUCCESS') {
    // single quoted values will be expanded when passed to injectEnvVars
    _env['hipchat_message'] = 'Server build succeded: <a href="https://$SERVER_NAME/">$SERVER_NAME</a> (<a href="$BUILD_URL">Job</a>)'
} else {
    ...
}

injectEnvVars(_env)
'''
evaluate(new File("/home/jenkins/workspace/devops/inject_env_vars.groovy"))

These should be run as System Groovy script.

I hope this could be someday implemented as a plugin.