On Software

Android + CMake is much easier in ndk-r5b

2011-06-08T14:35:00.000-07:00

About a year or two ago we struggled to make regular CMake usage with Android easy, time has changed and with the newer Android ndk releases things are straightforward.

Bellow is content of android_env.sh script. To use it with CMake do following

open command line + cd to your project directory with CMakelist.txt
. ./android_env.sh
ccmake .

#!/bin/sh

export NDK=/home/vasek/android-ndk-r5b
SYSROOT=$NDK/platforms/android-9/arch-arm

MIDDLE=toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/
PREF=arm-linux-androideabi-

export CC="$NDK/$MIDDLE/${PREF}gcc --sysroot=$SYSROOT"
export CXX="$NDK/$MIDDLE/${PREF}g++ --sysroot=$SYSROOT"
export LD="$NDK/$MIDDLE/${PREF}ld --sysroot=$SYSROOT"
export CPP="$NDK/$MIDDLE/${PREF}cpp --sysroot=$SYSROOT"
export AS="$NDK/$MIDDLE/${PREF}as --sysroot=$SYSROOT"
export OBJCOPY="$NDK/$MIDDLE/${PREF}objcopy --sysroot=$SYSROOT"
export OBJDUMP="$NDK/$MIDDLE/${PREF}objdump --sysroot=$SYSROOT"
export STRIP="$NDK/$MIDDLE/${PREF}strip --sysroot=$SYSROOT"
export RANLIB="$NDK/$MIDDLE/${PREF}ranlib --sysroot=$SYSROOT"
export CCLD="$NDK/$MIDDLE/${PREF}gcc --sysroot=$SYSROOT"
export AR="$NDK/$MIDDLE/${PREF}ar --sysroot=$SYSROOT"

Shortening the debug phase

2011-01-13T03:25:00.000-08:00

One way to be able to quickly slash bugs in embedded software projects is to have good information about the context bugs appear. My colleagues tried to summarize their experience from large software projects in automotive to the following paper -Ten steps to efficient bug analysis.

"This whitepaper describes approach allowing comprehensive analysis of bugs offline which dramatically boosts efficiency of development team during bug fixing phase."

Content is following :

Make log analysis user friendly
Mark bug in the log
Restore system state automatically from the log
Visualize performance and memory issues
Store call stacks in the log
Generate KPI reports directly from logs
Logging outside the application process
Unify logging mechanism
Optimize CPU overhead of logging
Compress logging data

Working with IBM Jazz via Python through REST api

2010-10-06T02:05:00.000-07:00

Have you worked with RTC, RQM or any other IBM Jazz based tool? Maybe you needed to import or export data via your own script?

There are some examples in Java and C# showing how to interact with RQM in this document. My preference was python. This post is a little example how to use python to interact with RQM server.

Motivation
It would be nice to have a python RQM client working like this.

from jazzclient import JazzClient

jazz = JazzClient('https://servername:9443/jazz','username','password')
workItemXml = jazz.getWorkItem('3934')
jazz.changeProgress('3934', '90')

Interacting with the REST interface is easy via the httplib2 python module.

1. Authentication
consists of 3 steps (details on jazz.net here)

requesting a "restricted resource"
supplying credintials and getting togetherthe session ID
requesting the "restricted resource" again, this time we should get it and confirm that our session is authenticated

All the authentication is done during construction, see bellow.

import urllib
import httplib2
from xml.dom.minidom import parse, parseString
from exceptions import Exception

class JazzClient(object):
    def __init__(self, server_url, user, password):
        self.base_url = server_url

        self.http = httplib2.Http()
        self.http.follow_redirects = True
        self.headers = {'Content-type': 'text/xml'}

        #1) before authentification one needs to go first to a "restricted resource"
        resp, content = self.http.request( self.base_url + "/oslc/workitems/1.xml", 'GET', headers=self.headers)
        #TODO sometimes returns capital X-
        #TODO check before if the key is in dictionary, if not something is wrong as well
        if resp['x-com-ibm-team-repository-web-auth-msg'] != 'authrequired':
            raise Exception("something is wrong seems the server doesn't expect authentication!")


        self.headers['Cookie']= resp['set-cookie']
        self.headers['Content-type'] = 'application/x-www-form-urlencoded'

        #2 now we can start the authentication via j_security_check page
        resp, content = self.http.request(self.base_url+'/j_security_check' , 'POST', headers=self.headers,
                    body=urllib.urlencode({'j_username': user, 'j_password': password}))

        #TODO check auth worked fine, if not throw exception
        #3 get the requested resource - finish the authentication
        resp, content = self.http.request( self.base_url + "/oslc/workitems/1.xml", 'GET', headers=self.headers)

2. Getting a first "WorkItem" in XML (getWorkItem)

Once we are authenticated it's just simple (follow the API documentation ). getWorkItem is method of JazzClient class.

def getWorkItem(self, itemNumber):

        self.headers['Content-type'] = 'text/xml'
        resp, content = self.http.request(self.base_url+'/oslc/workitems/'+ itemNumber +'.xml', 'GET', headers=self.headers)
        if resp.status != 200:
            raise Exception("JazzClient responce status != 200 !!!")
        return content 

3. Changing progress field in a WorkItem (changeProgress )

Now a small example showing change of progress field in a WorkItem. Again following method belongs to above JazzClient class.

    def changeProgress(self, itemNumber, progress):
        changeRequest = """
                <oslc_cm:changerequest xmlns:atom="http://www.w3.org/2005/Atom" xmlns:calm="http://jazz.net/xmlns/prod/jazz/calm/1.0/" xmlns:dc="http://purl.org/dc/terms/" xmlns:jd="http://jazz.net/xmlns/prod/jazz/discovery/1.0/" xmlns:jp="http://jazz.net/xmlns/prod/jazz/process/1.0/" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:oslc_cm="http://open-services.net/xmlns/cm/1.0/" xmlns:oslc_disc="http://open-services.net/xmlns/discovery/1.0/" xmlns:oslc_qm="http://open-services.net/xmlns/qm/1.0/" xmlns:oslc_rm="http://open-services.net/xmlns/rm/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/">
                <rtc_cm:progress>%s</rtc_cm:progress>
                </oslc_cm:changerequest>"""% (progress)                
        self.headers['Content-type'] = 'text/xml'
        resp, content = self.http.request(self.base_url+'/oslc/workitems/'+ itemNumber +'.xml' , 'PUT', headers=self.headers, body=changeRequest)
        if resp.status != 200:
            raise Exception("JazzClient.responce status != 200 !!!")

        return content

Conclusion
With the REST api documentation and bit of python it is fun to interact with Jazz servers.
The above code of JazzClient class were tested so just copy&paste to use it.

Links

RTC REST API v2.0
RQM API - Data model of RQM
RQM Using Poster - howto access RQM using Firefox Poster plugin d
OSLC Workshop - Consumer & Producer labs
RQMURLUtility - example java client for REST API including source code

Some links expect registration (free).

Building native libraries for Android using CMake

2010-01-31T07:34:00.000-08:00

Some time ago we started to use CMake build scripts for the Android platform since most of our cross-platform projects are based on CMake.

Maybe this style of building is also of your preference.
If so, here is a little how-to.

CMake and Android NDK need tobe installed
create a gcc-android file (copy paste from bellow)
create a symlink to gcc-android called g++-android
create android-build-env file on your system (see bellow)
adjust ANDROID_GCC_WRAPPERS variable (in android-build-env!) to point to directory where gcc-android is
run " . ./android-build-env" to set proper environment to tell CMake what tools to use for project building
generate project files from CMake file (e.g. using ccmake)
generated project is ready to build the binaries using gcc-android gcc wrapper.

Scripts were tested with Eclipse and make on (Ubuntu) Linux.

File - android-build-env


export ANDROID_NDK_ROOT="$HOME/android-ndk-1.6_r1"
 export ANDROID_GCC_WRAPPERS="!! PLS CHANGE !! PATH_TO_gcc-android script"
 export PATH="$ANDROID_GCC_WRAPPERS:$ANDROID_NDK_ROOT/build/prebuilt/linux-x86/arm-eabi-4.2.1/bin/:$PATH"
 export CROSS_PREFIX="arm-eabi-"
 export SYSROOT="$ANDROID_NDK_ROOT/build/platforms/android-4/arch-arm"
 
 # The config is specified in $ANDROID_NDK_ROOT/build/toolchains/arm-eabi-4.2.1/setup.mk
 # and $ANDROID_NDK_ROOT/build/core/build-*.mk
 
 export MAKE="make"
 
 # Use android wrappers
 export CC="gcc-android"
 export CXX="g++-android"


  export LD="deliberatey-undefined-do-not-use-directly"
  export CPP="${CROSS_PREFIX}gcc -E"
 export AS="${CROSS_PREFIX}as"
  export OBJCOPY="${CROSS_PREFIX}objcopy"
 export OBJDUMP="${CROSS_PREFIX}objdump"
  export STRIP="${CROSS_PREFIX}strip"
 export RANLIB="${CROSS_PREFIX}ranlib"
 export CCLD="${CROSS_PREFIX}gcc"
 export AR="${CROSS_PREFIX}ar"

File - gcc-android

#!/bin/bash
# Android gcc/g++ wrapper
#
# As android uses completely hacked and gutted gcc, it also
# uses -nostdlib, so for the linking phase we must explicitly
# specify crtbegin, crtend and friends. => We can't just do
# with LDFLAGS. :-(
#
# set DRY_RUN=1 for dry run
# set V=1 for verbose run
V=1

if [ -z $CROSS_PREFIX ]; then
    echo "Source android-build-env first!"
    exit 1
fi

REAL_CC="${CROSS_PREFIX}gcc"
REAL_CXX="${CROSS_PREFIX}g++"
# Inspired by ndk-wrappers:COMMON_FLAGS="--sysroot=$SYSROOT -march=armv5te -mtune=xscale -msoft-float -mthumb-interwork -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ -DANDROID"

CFLAGS="$COMMON_FLAGS"

CXXFLAGS="$COMMON_FLAGS" -fno-exceptions -fno-rtti"

TARGET_LIBGCC=`${CROSS_PREFIX}gcc -mthumb-interwork -print-libgcc-file-name`

# Test if we're wrapping gcc or g++
THIS_SCRIPT=`basename "$0"`

if [[ "$THIS_SCRIPT" =~ 'gcc' ]]; then
    WRAPPED="$REAL_CC"
    COMPILE_FLAGS="$CFLAGS"
elif [[ "$THIS_SCRIPT" =~ 'g++' ]]; then
    WRAPPED="$REAL_CXX"
    COMPILE_FLAGS="$CXXFLAGS"
else
    echo "Wrong script name - matches neither '^gcc' nor '^g++'."
    exit 2
fi

# Determine what mode (link, compile, nothing) we're in
LINK=1
COMPILE=0
SHARED=0
for ARG in "$@"; do
    case "$ARG" in
        # Preprocess-only (-E) and don't assemble (-S) mean the same as -c for us - the same CFLAGS are needed
        -c|-S|-E)
            LINK=0
            ;;

        *.c|*.cpp|*.c++|*.cxx)
            COMPILE=1
            ;;

# This is broken: gcc -c src.c -o object.o
#        *.o)
#            LINK=1
#            ;;

        -shared)
            SHARED=1
            ;;

        -static)
            echo "$0: Static executables not supported by this wrapper."
            exit 5
            ;;

        -v)
            LINK=0
            COMPILE=0
            break
            ;;
    esac
done

if [ $# -eq 0 ]; then
    COMPILE=0
    LINK=0
fi

function verbose_command
{
    if [ "$V" = "1" -o "$DRY_RUN" = "1" ]; then
        echo "$@"
    fi

    if [ "$DRY_RUN" != "1" ]; then
        "$@"
    fi
}

# Compile source
function compile
{
    # That's easy
    verbose_command "$WRAPPED" $COMPILE_FLAGS "$@"
}

# Link a binary
function link
{
    # TODO: Separate libraries, sources and other arguments
    # Idea: read args till hitting sources -> that's USER_LIBS
    # rest is USER_OBJECTS, ignore USER_ARGS
    USER_LIBS=""
    USER_OBJECTS=""
    USER_ARGS=""

    OBJECTS_HIT=0
    for ARG in "$@"; do
        if [ $OBJECTS_HIT -eq 0 ]; then
            # Test if ARG is a lib
            if [[ "$ARG" == *.o ]]; then
                # is an object
                OBJECTS_HIT=1
                USER_OBJECTS="$USER_OBJECTS $ARG"
            else
                # a lib or an early param
                USER_LIBS="$USER_LIBS $ARG"
            fi
        else
            # We already reached objects, stuff all into USER_OBJECTS
            USER_OBJECTS="$USER_OBJECTS $ARG"
        fi
    done

    # Mess starts here
    if [ $SHARED -eq 1 ]; then
        # TODO: Remove -shared from args

        # Building a shared library
        LD_PREFIX="--sysroot=$SYSROOT -nostdlib -Wl,-shared,-Bsymbolic"
        LD_BEGIN="$SYSROOT/usr/lib/libc.so $SYSROOT/usr/lib/libm.so"
        # user objects
        # user dyn. libs
        LD_END="-Wl,--no-undefined -Wl,-rpath-link=$SYSROOT/usr/lib $TARGET_LIBGCC"

        verbose_command "$WRAPPED" $LD_PREFIX $USER_OBJECTS $USER_LIBS $LD_BEGIN $LD_END $USER_ARGS
    else
        # Building dynamic executable
        LD_PREFIX="--sysroot=$SYSROOT -nostdlib -Bdynamic -Wl,-dynamic-linker,/system/bin/linker -Wl,--gc-sections -Wl,-z,nocopyreloc"
        # STLport static lib explodes here, needs to come after objects (??)
        LD_BEGIN="$SYSROOT/usr/lib/libc.so $SYSROOT/usr/lib/libm.so $SYSROOT/usr/lib/crtbegin_dynamic.o"
        # user objects
        # user dyn. libs
        LD_END="-Wl,--no-undefined -Wl,-rpath-link=$SYSROOT/usr/lib $TARGET_LIBGCC $SYSROOT/usr/lib/crtend_android.o"

        verbose_command "$WRAPPED" $LD_PREFIX $LD_BEGIN $USER_OBJECTS $USER_LIBS $LD_END $USER_ARGS
    fi
}

if [ $COMPILE -eq 0 -a $LINK -eq 0 ]; then
    # No special handling needed, call $WRAPPED
    verbose_command "$WRAPPED" "$@"
fi

# TODO: support link+compile
if [ $COMPILE -eq 1 -a $LINK -eq 1 ]; then
    echo "$0: Link and compile in one step is not supported by this wrapper."
    exit 4
fi

if [ $COMPILE -eq 1 ]; then
    compile "$@"
elif [ $LINK -eq 1 ]; then
    link "$@"
else
    echo "$0: Internal error: Uknown compiler mode."
    exit 3
fi

pygtkie - Internet Explorer in your pygtk application

2009-05-07T02:43:00.000-07:00

pygtkie is a small component for embedding Internet explorer to pygtk applications ( on Windows of course) using python language.

The picture bellow shows what you can do with pygtkie. Among other things a start up page for a program. If user clicks on a specific link a custom python code can be called.

Please see code examples on pygtkie.

eliminate the monkey work

2009-04-19T12:16:00.000-07:00

Sometimes people suggest to hire a student (== not so expensive technical person) to do some repetitive a tedious tasks. Eg. testing a specific module, pre-chewing certain complex document etc.

It's a temptation, especially when you have a big enough budget and a good will to help your overloaded software team. But at the end it maybe has the opposite effect.

A few disadvantages which just come to my mind:

if the hard monkey job is done by somebody else you don't even start to think how to eliminate it (eg. module is automaticaly testable only if it's designed the right way)
repetitive work is no fun - bad mood doesn't increase the productivity of your team
the more people in the team, the more time spent in meeting, communication, management, administration...

In software factories most of the repetitive and boring work can be eliminated. Some repetitive tasks are harder to remove some are just easy. Complex documents can be managed by requirement management tools, most of the testing can be decomposed in a way that it could be run automatically, integration test can be semi-automated

If one succeeds in eliminating this repetitive tasks he/she is rewarded by having a smaller team which is more flexible, faster and most probably the deliverable will contain less bugs.

Let me put it simple: "Monkey tasks must be shared by the whole team, otherwise the senior people are not motivated to eliminate them."

Data2l 1.0 - Management of complex data structures

2009-04-08T13:37:00.000-07:00

We finally released Data2l 1.0.
It helped us a lot during the last 3 years in understanding, generating, patching large "embedded" (also called "read-only" or "flatfiles") databases.
After those years of internal usage and internal development we cut off all the proprietary parts and released a general tool.

If you manage a large read-only database - typically in embedded system like car navigation - have a look it might save you a lot of work.

Using couple of types like Structure, Alternative, Array and Basic ( word, byte, dword ...) and using really simple "functional" language for relation definition we can describe really complex binary structures.

Once structure is described it's so easy do anything with it .. generate new data, inspect old data, use SAX and DOM like access, transform data or generate a specialized maintenance code. We used generated C++ code and this way we could work with a large binary data with quick and ease.

Repeat after me: "Singleton is my enemy"

2009-03-19T14:52:00.000-07:00

Some people call singletons controversial some call them evil. I'd say the second group consists of those who pray to OOP.

What are the reasons why they are evil especially in large scale software design?

they cripple locality - singleton is equal to a global variable - it's functionality can suddenly appear anywhere in the program. This decreases dramatically understandability of the system and often forces you to use new and new singletons, especially if management is pushing for new features and there is no time for refactoring.
they increase number of states in the system - singletons don't have parametrized constructors - that enforces to use "set" methods which add the necessary context to singletons. But that means that we need to check in singleton methods that the context is correctly set. So again in all the functions we have couple of more "if" statements, if we have "if" statements we probably have to report failure return codes. "hello! here is a singleton which doesn't know its context!".
Little by little we need to add more and more code which takes care about the state of the system. This doesn't happen during weeks or months. This happens during years of software maintenance. People coming and leaving .. managements is pushing ... do you know the story ? :-)
they are difficult to release - singleton is usually not owned by any other object. Those who are not owned are then released in compiler specific defined order. This is not what we want to have. When there are more singletons they are normally that entangled that graceful release of such objects is not possible. If one cannot gracefully shutdown it's system resource leak analysis is difficult.

There are many reasons and explanations above three are the first which come to my mind just yahoo "why singletons are bad" or something similar to see the consensus.