SoapUI / REST / JSON / variable namespaces

As a developer of large Web applications, I'm used to relying on a few proven test strategies:

• Unit tests with JUnit / Cobertura and DOH / JSCoverage;
• Functional and Integration tests with Selenium 2.

When I joined my current development team to work on the project TradeInsight, I was introduced to SoapUI for the functional tests. I like it's ability to convert JSON to XML, enabling then the writing of XPath match assertions.

There's one little caveat related to the conversion and the XPath assertions:

• When the server response contains a single JSON object, the conversion introduce a namespace into the generated XML.
• And this namespace depends on the server address.

The following figures show the original JSON response and its conversion to XML with the inferred namespace.

JSON response produced by a REST service.

Transformation into a XML payload with an inferred namespace.

This automatic mapping to namespace server dependent does not allow to write server-agnostic code if you follow the suggested solution!

The following figures show a XPath match expression as documented in the SoapUI training materials. Sadly, running it against an XML with a namespace, this error is reported:

XPathContains assertion failed for path [//startDate/text()] : Exception missing content for xpath [//startDate/text()] in Response.

Simple XPath expression with the corresponding error message.

Corrected XPath expression as suggested, now server dependent :(

 A simple solution consists in replacing the specified namespace with the meta-character '*', which match any namespace. As all elements of the XML document are under the scope of the inferred namespace, it's important to prefix all nodes of the XPath expression with '*:', as illustrated by the following figure.

Use of the '*' prefix to produce assertion server independent.

I hope this helps.

A+, Dom

Un nouveau chapître

Il y a quelques 18 moins, j'ai choisi de devenir un entrepreneur à temps plein, consacrant mon temps et mes ressources financières au développement du service AnotherSocialEconomy. Cela a été une expérience très riche en enseignements.

Ceux qui me connaissent savent à quel point il était important que je sois vraiment maître de mon destin professionnel :

• Développer un outil qui offre une vraie valeur à son groupe d'utilisateurs.
• Baser les décisions sur les faits et leurs conséquences sans laisser places aux interférences politiques (tellement immobilisantes dans les grandes entreprises).
• Utiliser la technologie pour servir d'abord les usagers, puis pour construire les services et faciliter leur développement. En ce sens, la méthodologie Lean Startup et son principe Build-Measure-Learn offre un excellent cadre de travail.
• Utiliser la méthodologie Agile (sans oublier les outils de XP) pour développer une application de grande envergure.

Avec mon associé Steven, cela a été très intéressant de définir mes propres objectifs, mon organisation de travail, et diriger le développement de l'entreprise.

Mais il y a un mois, principalement à cause de l'absence d'une perspective de revenus stables dans un futur proche, je me suis mis à la recherche de contrats et de postes permanents. C'est sûr que c'est déchirant, mais l'outil est en production et je peux le faire évoluer dans mon temps libre. En appliquant donc un principe de Lean une fois de plus ("Failure to change is a vice!", Hiroshi Okuda, Président de Toyota Motor Corp.), je replonge du côté salarié.

Finalement, après plusieurs entretiens, mon choix s'est porté sur la compagnie MEI qui développe une plate-forme Web pour aider ses clients (des producteurs de biens de consommation emballés--consumer packaged goods) à suivre leurs campagnes de promotion. Jusqu'il y a deux ans, MEI offrait principalement son service aux grands manufacturiers. Maintenant, dans une offre SAAS, MEI développe une nouvelle version pour les producteurs petits et intermédiaires, sous la marque TradeInsight. La synergie entre mon expérience (Java, JavaScript, cloud, mobile, Agile) et l'équipe est indéniable.

Au fait : MEI embauche encore !

L'aventure d'AnotherSocialEconomy continue, à temps perdu de mon point de vue et dans mes interactions avec les groupes de la communauté technique de Montréal (NewTech, Android, NodeJS, etc.). Pour plus d'informations quant aux partenariats possibles, veuillez prendre contact avec Steven.

A+, Dom

OAuth authorization handling in a Android application

Note : This post is part of the series "Lessons learned as an independent developer". Please refer to the introduction (in French) for more information. Cet article fait partie de la série intitulée « Leçons d'un développeur indépendant ». Au besoin, lisez mon introduction pour plus d'informations.

Context

AnotherSocialEconomy APIs are based on standards as much as possible: OpenID and OAuth for the authentication and authorization, HTTP-based REST interface (1, 2) for the communication protocol, JSON for the data payload format, etc.

This post is about setting up a Android application which get authorization tokens from a OAuth provider.

OAuth provider

There are many known OAuth providers like Netflix, Twitter, Facebook (coming to OAuth 2.0 soon), Yahoo!, Google, etc. If these providers are convenient, they don't offer much flexibility if some debugging is required.

For this experiment, I'm going to use Google App Engine Java and their OAuth support. For a complete walk-through, refer to Ikai Lan's post: Setting up an OAuth provider on Google App Engine, especially for the part which describes how to get the public and secret keys for your client application to sign communications with the provider.

OAuth client - Work flow

Strategy: Upon creation, the process checks if the authorization tokens have been saved as user preferences.

• If they are present, they are loaded to be used to sign each future communication with the application on the server.
• If they are missing, the OAuth work flow is triggered:
1. With the server application keys, a signed request is sent to get a temporary request token.
2. With this request token, a URL to the authentication page is required and an Intent is created to load the corresponding page in a browser. At this step, the application is stopped.
   
3. The user enters his credentials in the browser and grants access rights to the mobile application. The return URL has a custom format: ase://oauthresponse.
4. The mobile application, which has an Intent registered for that custom URL, is restarted and is given the return URL. A verification code is extracted from this URL.
5. The verification code is used to issue a signed request asking for the access tokens.
• The access tokens are saved as part of the user preferences only if she selected a 'Remember me' option.
  

Figure 1: Authorization work flow

Alternative: If the mobile application offers anonymous services, like browsing the list of registered stores in the case of AnotherSocialEconomy.com, it can be friendlier to delay the authorization verification.

OAuth client - Initiating the authorization process (1, 2, 3)

To simplify the application development, I have decided to use oauth-signpost, a library provided by Matthias Käppler who wanted a slick and simple way to access Netflix services.

Signpost is the easy and intuitive solution for signing HTTP messages on the Java platform in conformance with the OAuth Core 1.0a standard. Signpost follows a modular and flexible design, allowing you to combine it with different HTTP messaging layers.

Note that this library is also good to manage remotely Twitter accounts.

This section is about initiating the authorization process, which occurs if the application is not called by the application on the server (with the verification code, see next section) and if the OAuth token could not be found in the user preferences. This is the path with the steps {1, 2, 3} in Figure 1.

if (!justAuthenticated && Preferences.get(Preferences.OAUTH_KEY, "").length() == 0) {
    // Display the pane with the warning message and the sign in button
    setContentView(R.layout.main_noauth);

    // Update the 'Remember me' checkbox with its last saved state, or the default one
    final String saveOAuthKeysPrefs = Preferences.get(Preferences.SAVE_OAUTH_KEYS, Preferences.SAVE_OAUTH_KEYS_DEFAULT);
    ((CheckBox) findViewById(R.id.app_noauth_keepmeconnected)).setChecked(Preferences.SAVE_OAUTH_KEYS_YES.equals(saveOAuthKeysPrefs));

    // Attach the event handler that will initiate the authorization process up to opening the browser with the authorization page
    findViewById(R.id.app_noauth_continue).setOnClickListener(new OnClickListener() {
        @Override
        public void onClick(View v) {
            // Check if the 'Keep me connected' check box state changed and save its new state
            boolean keepMeConnected = ((CheckBox) findViewById(R.id.app_noauth_keepmeconnected)).isChecked();
            if (Preferences.SAVE_OAUTH_KEYS_YES.equals(saveOAuthKeysPrefs) != keepMeConnected) {
                Preferences.set(Preferences.SAVE_OAUTH_KEYS, keepMeConnected ? Preferences.SAVE_OAUTH_KEYS_YES : Preferences.SAVE_OAUTH_KEYS_NO);
            }
                    
            // Set up the OAuth library
            consumer = new CommonsHttpOAuthConsumer("<your_app_public_key>", "<your_app_secret_key>");

            provider = new CommonsHttpOAuthProvider(
                    "https://<your_app_id>.appspot.com/_ah/OAuthGetRequestToken",
                    "https://<your_app_id>.appspot.com/_ah/OAuthAuthorizeToken",
                    "https://<your_app_id>.appspot.com/_ah/OAuthGetAccessToken");
                    
            try {
                // Steps 1 & 2:
                // Get a request token from the application and prepare the URL for the authorization service
                // Note: the response is going to be handled by the application <intent/> registered for that custom return URL
                String requestTokenUrl = provider.retrieveRequestToken(consumer, "ase://oauthresponse");

                // Step 3:
                // Invoke a browser intent where the user will be able to log in
                startActivity(new Intent(Intent.ACTION_VIEW, Uri.parse(requestTokenUrl)));
            }
            catch(Exception ex) {
                Toast.makeText(Dashboard.this, R.string.app_noauth_requesttoken_ex, Toast.LENGTH_LONG).show();
                Log.e("Dashboard no auth", "Cannot initiate communication to get the request token\nException: " + ex.getClass().getName() + "\nMessage: " + ex.getMessage());
            }
        }
    });
}

Figure 2 below illustrates the pane main_noauth displaying the warning message and the action button, and figure 3 shows the authorization page as provided by Google for the hosted applications on App Engine.

Figure 2: Pane displayed if application not yet authorized

Figure 3: Google authorization page

Whatever action the user takes, the application is going to be called with the URL ase://oauthresponse. The next section covers this work flow path.

OAuth client - Processing the authorization (4, 5)

The application is registered with an Intent associated to the scheme ase and the host oauthresponse. The labels themselves are not important, only their uniqueness and the correspondence with the return URL specified at Step 2.

<intent-filter>
    <action android:name="android.intent.action.VIEW"/>
    <category android:name="android.intent.category.DEFAULT" />
    <category android:name="android.intent.category.BROWSABLE"/>
    <data android:scheme="ase" android:host="oauthresponse"/>
</intent-filter>

The following code snippet implements the steps 4 and 5 as described in Figure 1.

private boolean checkOAuthReturn(Intent intent) {
    boolean returnFromAuth = false;
    Uri uri = intent.getData();

    if (uri != null && uri.toString().startsWith("ase://oauthresponse")) {
        // Step 4:
        // Get the request token from the Authentication log in page
        String code = uri.getQueryParameter("oauth_verifier");
            
        try {
            // Step 5:
            // Get directly the access tokens
            provider.retrieveAccessToken(consumer, code);
            returnFromAuth = true;
                
            // Persist the tokens
            if (Preferences.SAVE_OAUTH_KEYS_YES.equals(Preferences.get(Preferences.SAVE_OAUTH_KEYS, Preferences.SAVE_OAUTH_KEYS_DEFAULT))) {
                Preferences.set(Preferences.OAUTH_KEY, consumer.getToken());
                Preferences.set(Preferences.OAUTH_SECRET, consumer.getTokenSecret());
            }
        }
        catch(Exception ex) {
            Toast.makeText(Dashboard.this, R.string.app_noauth_accesstoken_ex, Toast.LENGTH_LONG).show();
            Log.e("Dashboard no auth", "Cannot complete communication to get the request token\nException: " + ex.getClass().getName() + "\nMessage: " + ex.getMessage());
        }
    }
       
    return returnFromAuth;
}

The Dashboard class definitions are available in a gist on GitHub. This gist contains also a wrapper of the SharedPreferences class, the application manifest with the declaration of the Intent for the custom return URL, and the layout definition of the pane with the warning and the sign in button.

OAuth Client - The quirks

My Android application is very simple and is configured with the launch mode singleTop. As such, if the system does not destroy the application when the code starts an activity to browse the Authentication service URL, the invocation of the ase://oauthresponse URL by the browser should trigger a call to the onNewIntent() method. It never happened during my tests and on my phone... Every time, the application is recreated and a call to onCreate() is issued. So both functions delegate to the helper checkOAuthReturn().

@Override
protected void onNewIntent(Intent intent) {
    checkOAuthReturn(intent);
}

In this example, I've decided to select the view to associate to the first screen of the application according to the knowledge of the OAuth access token (read from the user preferences or retrieved dynamically thanks to the verification code coming with the ase://oauthresponse URL). The following snippet illustrates this flow. In some occasions, it can be better to start a separate activity if the main pane is instrumented to disable the triggers to protected actions. This approach with a separate activity is also better for the portability.

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    Preferences.setPreferenceContext(PreferenceManager.getDefaultSharedPreferences(getBaseContext()));

    boolean justAuthenticated = checkOAuthReturn(getIntent());
        
    if (!justAuthenticated && Preferences.get(Preferences.OAUTH_KEY, "").length() == 0) {
        setContentView(R.layout.main_noauth);

        // Instrumentation of the pane to initiate the authorization process on demand
        // ...
    }
    else {
        setContentView(R.layout.main);
    }
}

I hope this helps.
A+, Dom

Partage d'expérience dans le développement d'applications Android

Contexte

Note : Cet article est le premier d'une série intitulée « Leçons d'un développeur indépendant ». Le niveau de la discussion dans cet article est général. Les articles suivant iront plus en profondeur et seront illustrés de bouts de code.

Dans le cadre de mon entreprise AnotherSocialEconomy.com, j'ai eu l'occasion de mettre en œuvre plusieurs bonnes pratiques et je vais en partager quelques unes ici. Je vais notamment me concentrer sur les développements autour de l'application cliente Android, depuis l'identification des usagers jusque l'émission de notifications asynchrones.

AnotherSocialEconomy.com, ou ASE, offre un service connectant consommateurs et détaillants :

• Les consommateurs à la recherche d'un produit ou service n'ont qu'à décrire leur demande depuis l'un des multiples points d'entrée de l'application : une page Web faite pour AdWords, le site de ASE ou affilié, la page de l'application Facebook, un message direct depuis Twitter, etc.
• Les détaillants participants sont notifiés des demandes en fonction de leurs préférence. Les détaillants sont libres de faire une ou plusieurs propositions en fonction de leur disponibilité.
• Au fur et à mesure que les propositions sont composées, les consommateurs en sont notifiés et peuvent à tout moment les décliner ou les confirmer.
• Les confirmations sont notifiés aux détaillants qui réservent alors le produit ou le service pour le consommateur.
• Ce dernier n'a plus qu'à payer et à en prendre possession.

Pour faire simple : ASE connecte les consommateurs avec les détaillants qui ont les produits ou services qu'ils recherchent en inversant le processus de recherche.

Le moteur de ASE est présentement codé en Java et est hébergé sur l'infrastructure Google App Engine. Dans la suite de cet article, pour généraliser le propos, le moteur de ASE est référencé en tant qu'application serveur.

La gestion des utilisateurs sur le serveur

Depuis le départ, le service des usagers de l'application serveur repose sur OpenID. Avec OpenID, l'identification des utilisateurs est confiée à des services tiers de confiance (Yahoo!, Google, AOL, etc.) sans que l'application serveur ne voit le mot de passe des utilisateurs, seulement leur identifiant OpenID. Ce mode de gestion règle aussi plusieurs problèmes :

• Les utilisateurs n'ont pas à créer un énième compte pour le service ASE.
• Le serveur est moins à risque car il n'y a pas de mots de passe enregistré là.
• La gestion des sauvegardes est plus simple (toujours parce qu'il n'y a pas de mots de passe).
• En cas de bris de leur mot de passe, les utilisateurs peuvent assurément mieux s'appuyer sur les services de leur fournisseur OpenID que sur les miens :)

Plus tard dans le cycle de développement, notamment parce qu'il a s'agit de développer une application Facebook, les mécanismes d'identification de Facebook, ceux de Twitter et de ceux de Microsoft Live ont été intégrés à l'application serveur. Des trois, le mécanisme de Twitter est le plus standardisé (OAuth), donnant aussi accès aux données de l'utilisateur du service. Mais tous ont été intégrés de manière à agir comme des services OpenID.

OpenID est un bon système d'identification pour une application cliente Web. Avec les restrictions de sécurité des navigateurs (SSL et sandbox), une fois que l'identité de l'utilisateur est confirmée par un fournisseur OpenID, tant que cette identité reste associée à la session Web, l'envoi de données vers les navigateurs reste protégé.

Par contre, quand l'application cliente est native (sur un ordinateur ou sur un téléphone mobile), il n'est pas possible de s'appuyer sur un mode de session Web robuste comme celui des navigateurs. Aussi une application malicieuse pourrait intercepter l'identifiant de session et s'en servir à l'insu de l'usager. Pour se prémunir contre cette attaque, il est souhaitable d'utiliser OAuth qui signe chaque échange entre l'application cliente et le serveur, rendant caduc l'utilisation de l'identifiant de session Web.

L'authentification des usagers sur le client

Chaque téléphone Android est associé à un utilisateur. Si la carte SIM de l'opérateur téléphonique est changée, les données de l'utilisateur précédent ne sont plus accessibles. Chaque application à accès à son propre espace de stockage protégé, mais l'utilisateur peut réclamer cet espace à tout instant. Ce n'est donc pas une solution de stockage à long terme.

Dans le modèle d'authentification OAuth, les échanges de données sont signés par l'application cliente grâce à un jeton émis par l'application serveur. Grâce à la signature, l'application serveur est assurée de l'identité de l'utilisateur à chaque échange de données.

Pour avoir un jeton, le protocole à observer par l'application cliente est relativement simple :

• Émettre une requête pour recevoir un premier jeton dit d'accès.
• Ce jeton est utilisé pour initier un appel vers une page d'autorisation.
• L'application serveur présente alors une page d'identification où l'utilisateur doit, s'il n'est pas déjà authentifié, entrer son identifiant et son mot de passe, puis accepter que l'application cliente accède aux données qui sont gérées par l'application serveur.
• L'application serveur retourne un second jeton attestant de l'acceptation par l'utilisateur de l'accès aux données. Ce jeton a une durée de vie limitée.
• Ce second jeton peut être utilisé pour obtenir deux jetons (clé publique et clé secrète) qui permettront à l'application cliente de signer les échanges de données de telle sorte que l'application serveur les associera à l'utilisateur concerné.
• Souvent ces deux jetons ont une grande durée de vie (pas d'expiration dans le cas de Twitter), et peuvent donc être sauvegardés par l'application cliente pour signer de manière transparente tous les futurs échanges.
• Il faut cependant tenir compte que l'utilisateur peut révoquer ces deux jetons n'importe quand, ou qu'ils peuvent expirer n'importe quand (à cause d'un changement de stratégie du côté de l'application serveur, par exemple) aussi il faut être prêt à exécuter le processus pour obtenir deux nouveaux jetons à n'importe quel moment.

Il est important de noter que la sauvegarde des jetons d'authentification doit être très sécuritaire. Il n'est pas acceptable de les sauvegarder dans un simple fichier texte situé  sur une carte d'extension mémoire par exemple. Si le risque d'accès à ces jetons est trop grand, il faut mieux rejouer le scénario ci-dessus pour obtenir un nouveau jeu de jetons.

Au moment où j'écris cet article, le matériel de la série Samsumg S et la tablette Motorola Xoom ont des systèmes de fichiers encryptés. À ma connaissance, même Android 3.1 n'offre toujours pas de solution bas niveau de sécurité maximale...

La réception des notifications asynchrones

Si de plus en plus de fondeurs de silicium mettent l'accent sur la puissance du processeur central (Qualcomm) et leur nombre (NVidia vient d'annoncer un Tegra avec 4 cœurs), si l'augmentation de la bande passante (de HPSA+ à LTE par exemple) permet des échanges de données de plus en plus rapide même loin de tout réseau informatique, la capacité énergétique des téléphones portables modernes reste leur point faible. Par le passé, j'ai eu des téléphones Nokia et Sony Ericsson capables de rester en veille plus d'une semaine. Maintenant, je dois brancher mon téléphone HTC Desire chaque soir, et cela même avec une navigation somme toute restreinte !

Dans ces conditions, maintenir une application éveillée pour pouvoir interroger l'application serveur à intervalles réguliers (technique dite de polling) est à proscrire.

Il y a deux ans, en développant une application pour la plate-forme BlackBerry 5, j'ai utilisé la technique suivante :

• L'application client sur le téléphone écoutait un certain nombre de messages du système (changement de type de réseau, perte du réseau, etc.) et les colligeait dans une base de données interne.
• C'était l'application serveur qui décidait du moment de transmission de ces données statistiques en envoyant un SMS à chaque téléphone.
• À la réception de ce SMS, l'application client ouvrait une connexion HTTP pour transmettre en rafale ses données colligés.
• Une fois l'ensemble de données rapatriés de chaque téléphone, l'application serveur établissaient des rapports de couverture pour l'opérateur.

Depuis la version 2.2, il existe le protocole AC2DM: Android Cloud to Device Messaging. Quand une application cliente configurée pour AC2DM s'initialise, elle doit s'enregistrer auprès du serveur local AC2DM et reçoit en retour un identifiant d'enregistrement. C'est la responsabilité de l'application cliente d'envoyer cet identifiant à l'application serveur pour que celle-ci ait la clé pour envoyer les notifications asynchrones à cette application cliente, et à elle seule.

Quelque part, l'approche du AC2DM est semblable à ma méthode d'activation par SMS. Il se peut même qu'elle utilise en sous main cette technique ;) La principale différence réside dans l'aspect service : avec AC2DM, l'application cliente n'a pas à rester active pour recevoir les notifications, c'est le serveur local de notifications qu'il l'activera au besoin.

L'application pour les consommateurs

L'application cliente pour les consommateurs doit offrir plusieurs fonctionnalités :

• recevoir les notifications concernant les demandes et les propositions en attente
• gérer la liste des demandes et propositions en attente
• créer de nouvelles demandes
• avec un accès au carnet d'adresses du téléphone pour pouvoir inclure ses « amis » en copie des demandes
• avec un accès au système de localisation géographique du téléphone pour faciliter la création des demandes
• modifier ou annuler des demandes en attente
• confirmer ou annuler des propositions en attente

Le principal objectif de l'application cliente sur les téléphones mobiles est le relais des notifications de mise-à-jour de demande, en réaction à la réception de nouvelles propositions ou de modifications de proposition de la part de détaillants. En quelques « clics », l'utilisateur doit pouvoir accéder rapidement au détail de la demande concernée, au détail de la proposition et à des informations sur le magasin ou bureau du détaillant. Pour faciliter cet accès, la plupart des informations sont sauvegardées sur le téléphone au fur et à mesure qu'elles sont requises. Pour garder une structure proche du modèle de données produit par l'application serveur, le stockage utilisé est le service de base de données interne du mobile (SQLite sur Android, par exemple).

L'application pour les détaillants

L'application cliente pour les consommateurs doit offrir plusieurs fonctionnalités :

• recevoir les notifications de nouvelles demandes
• créer et gérer des propositions (possiblement avec un accès à la caméra pour scanner les codes barre)
• confirmer les livraisons
• gérer la liste des demandes et propositions en attente

Parce que les services offerts aux consommateurs sont très différents de ceux offerts aux détaillants, ils sont proposés dans deux applications différentes. Cela réduit les risques de confusion de contexte pour les utilisateurs agissant autant en tant que consommateur que détaillant.

À suivre...

Dans les prochains articles, je décrirai en détail les différentes implantations que j'ai réalisées. Il y a plusieurs techniques qui ne sont pas évidentes, comme celle gérant l'authentification avec OAuth, et j'imagine que cela sera utile à bien des développeurs ;)

A+, Dom

Google App Engine, scheduled tasks, and persisting changes into the datastore: the risk of a race condition

This post is about a race condition I've accidentally discovered and hopefully fixed. It occurred in App Engine and was generated by tasks I created for immediate execution...

Context

When I started developing in Java for Google App Engine, I decided to give a try with JDO, mainly because it is datastore agnostic (*). Operations managing my entities are organized in DAOs with set of methods like the followings.

public Demand update(Demand demand) {
    PersistenceManager pm = getPersistenceManager();
    try {
      return update(pm, demand);
    }
    finally {
        pm.close();
    }
}

public Demand update(PersistenceManager pm, Demand demand) {
    // Check if this instance comes from memcache
    ObjectState state = JDOHelper.getObjectState(consumer);
    if (ObjectState.TRANSIENT.equals(state)) {
        // Get a fresh copy from the data store
        ...
        // Merge old copy attributes into the fresh one
        ...
    }
    // Persists the changes
    return pm.makePersistent(demand);
}

I knew that changes are persisted only when the PersistenceManager is closed; closing it after an update is safe attitude. I decided anyway to separate the PersistenceManager instance management from the business logic updating the entity for clarity.

This decision offers the additional benefit of being able to share PersistenceManager instance with many operations. The following code snippet illustrates my point: a unique PersistenceManager instance is used for two entity loads and one save.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        // Get the identified demand (can come from the memcache)
        Demand demand = getDemandOperations().getDemand(pm, demandKey, ownerKey);

        // Check if the demand's location is changed
        if (command.contains(Location.POSTAL_CODE) || command.contains(Location.COUNTRY_CODE) {
            Location location = getLocationOperations().getLocation(pm, command);
            if (!location.getKey().equals(demand.getLocationKey())) {
                command.put(Demand.LOCATION_KEY, location.getKey());
            }
        }

        // Merge the changes
        demand.fromJson(command);

        // Validate the demand attributes
        ...

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);

        // Report the demand state to the owner
        ...
    }
    finally {
        pm.close();
    }
}

For my service AnotherSocialEconomy which connects Consumers to Retailers, the life cycle for a Demand is made of many steps:

• State open: raw data just submitted by a Consumer;
• State invalid: one verification step failed, requires an update from the Consumer;
• State published: verification is OK, and Demand broadcasted to Retailers;
• State confirmed: Consumer confirmed one Proposal; Retailer reserves the product for pick-up, or delivers it;
• State closed: Consumer notified the system that the transaction is closed successfully;
• State cancelled: ...
• State expired: ...

In my system, some operations take time:

• Because of some congestion in the environment, which occurs sometimes when sending e-mails.
• Because some operations require a large data set to be processed–like when a Demand has to be broadcasted to selected Retailers.

Because this time constraint and the 30 second limit, I decided to use tasks extensively (tasks can run for 10 minutes). In some ways, my code is very modular now, easier to maintain and test.

So I updated my code to trigger a validation task once the Demand has been updated with the raw data submitted by the Consumer. The code snippet shows the task scheduling in the context of the processDemandUpdate() method illustrated above.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        ...

        // Update the state so the entity is ready for the validation process
        demand.setState(State.OPEN);

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);

        // Create a task for that demand validation
        getQueue().add(
            withUrl("/_tasks/validateOpenDemand").
                param(Demand.KEY, demandKey.toString()).
                method(Method.GET)
        );
    }
    finally {
        pm.close();
    }
}

Issue

Until I activated the Always On feature, no issue has been reported for that piece of code: my unit tests worked as expected, my smoke tests were fine, the live site behaved correctly, etc.

Then the issue started to appear randomly: sometimes, updated Demand instances were not processed by the validation task anymore! A manual trigger of this task from a browser or curl had however the expected result...

For the task to be idempotent, the state of the Demand instance to be validated is checked: if set with open, the Demand attributes are checked with the result of the state being set with invalid or published. Otherwise nothing happens. With that approach, Demands already validated are not processed a second time...

What occurred?

• Without the Always On feature activated, because of the low traffic in my application, the infrastructure was delaying the process of the validation task a bit and it was executed once the request process finished.
• Thanks to that soft process serialization, the datastore update commanded by the instruction pm.close() had all chances to be completed before the start of the validation task!
• With the Always On feature activated, the infrastructure had much more chance to get one of the two other application instances to process the validation task... which could happen before the datastore update...
• As it started before the datastore update, the validation task found a task in the state set by the previous run of the task for this Demand instance: invalid or published. Then it exited without reporting any error.

Solutions

The ugly one:
Add a delay before executing the task with the countdownMillis() method.

// Create a task for that demand validation
        getQueue().add(
            withUrl("/_tasks/validateOpenDemand").
                param(Demand.KEY, demandKey.toString()).
                method(Method.GET).
                countdownMillis(2000)
        );
    }
    finally {
        pm.close();
    }
}

A tricky one:
Use memcache to store a copy of the Demand, which the validation will use instead of the reading it from the datastore. Because there's no warranty that your entity won't be evicted before the the run of the validation task, this is not a solution I can recommend.

The simplest one:
Move the code scheduling the code outside the try...finally... block. The task will be scheduled only if the updates of the Demand instance have been persisted.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        ...

        // Update the state so the entity is ready for the validation process
        demand.setState(State.OPEN);

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);
    }
    finally {
        pm.close();
    }

    // Create a task for that demand validation
    getQueue().add(
        withUrl("/_tasks/validateOpenDemand").
            param(Demand.KEY, demandKey.toString()).
            method(Method.GET)
    );
}

The most robust one:
Wrap everything withing a transaction. When a task is scheduled within a transaction, it's really enqueued when the transaction is committed.

Be aware that adopting this solution may require a major refactoring.

Conclusion

Now I understand the issue, I'm a bit ashamed of it. For my defense, I should say the defect has been introduced as part of an iteration which came with a series of unit tests. Before the activation of the Always On feature, it stayed undetected, and later it occurred only rarely.

Anyway, verifying the impact of all calls to external tasks before persisting any changes is one point in my review check list.

I hope this helps,
A+, Dom

--
Notes:
* These days, I would start my application with Objectify. This blog post summarizes many arguments I agree on too in favor to Objectify.

State of the AnotherSocialEconomy Initiative

When my partner Steven and I started our startup adventure a few years ago, our main goal was to demonstrate our ability to convert an idea into a live project. As we used our experience to build a viable product, we knew it would add value to our resume.

Over the months, the project evolved slowly:

• The core idea is: help the consumers who look for a specific product to find the retailer who has it in stock, and help retailers to connect with consumers online and drive them in-store. Our moto: the missing link between shopping online and buying offline.
• The proof-of-concept was made of screenshots, live Twitter accounts, and a piece of Python code connecting those accounts together. This material allowed us to be among the semi-finalist companies of TechCrunch50 in 2009!
• The first implementation of the engine connected consumers and retailers, each of them interacting with the system with direct messages (DMs), sent  from their own Twitter account. At that time, the tool was named Twetailer.
• Later, we figured out Twitter was too geeky and we added a connector to accept and generate e-mails. Since then, the engine has a XMPP (instant messaging) connector, another one for Facebook, and a plan for VOIP (with Twilio).
• At one point, we were approached to start an experiment for golfers: usually avid golfers have to spend a lot of time on the phone to get three buddies to play with and to book a tee-time. In two months, we created ezToff.com, developed an embeddable widget to ease the creation of a tee-off request, and developed a Web console for the golf course staff. The experiment was shut down because of a lack of traction...
• Recently, we started another experiment in the used car market, under the name AnotherSocialEconomy. Our market researches found that the average time to buy a used car is six weeks in Quebec. Typically, consumers start on the Web, grab listings, and call dealerships one after the other. In the Montreal area, there are 300,000 pre-owned cars bought per month: ⅓ from dealerships, ⅓ from wholesalers, and ⅓ from individuals. Dealerships control 45% of the market value.
• So far, this experiment has been a partial success: we get demands from consumers and forward proposals from used car sales people. We helped our first customer finding a car in only two weeks! But, as the dealership staff is not used to new technologies (sic), we manage the service for them and it's very time consuming...

What's next?

We are very happy with consumers trusting us. We work hard to continue to improve their experience, on our landing pages and in our communication by e-mail. The priority is to have them qualifying more their demands upfront.

Our focus right now is more on the retailer-side, in order to have sales people in the dealerships interacting with the system by e-mail too. If around 80% of them agree to work with us to serve our users, we have to prepare the proposal details and to reach them out for approval. For the business to scale, they should prepare and post the proposals themselves.

For now, we need more data to determine trends. This is a prerequisite for used car dealers to adopt our methodology. It is also possible that will lead to another pivot.

Lessons learned?

The first one is an obvious one now: nobody can be as committed as the founders! Since I left the company Compuware to become a full time entrepreneur, Steven and I have met many people we expected to work/partner with: a technology company CEO, a former manufacture owner and now real estate agent, a UX designer, a few VCs, a marketer, two successful startup founders, etc. If we sometimes got excellent feedback, none joined us.

The second one is related to the product development: two techies are not enough to make a great product! They can talk about their product at length, but they don't know how to convince decision makers. They need the help of a marketing genius!

Another one is related to the importance of the contact network. If you don't know the right people, very few will listen to you. Having a large address book or friends with deep pockets definitively helps a lot.

And the last one: developing a tool for the general public is difficult! Following the Lean Startup process can really help. Check Ash Maurya's blog, for example.

Technologies learned?

I continue to find Google App Engine an awesome environment. The recent addition of the Channel API which allows the back-end logic to push asynchronous notifications into Web consoles really improves the user experience. On the maintenance side, I appreciate the Java Remote API which simplifies the development of maintenance and data extraction tasks.

Web console side, I've started upgrading the code to Dojo 1.6 and its new HTML5 compliant syntax. I don't use the AMD loader yet, but I'm waiting for the one coming with 1.7. I have recently started to use Selenium 2 for my smoke tests and I really like it!

Mobile side, I wish to have more spare time to update my Android application for ezToff and to benefit from the Android Cloud To Device Messaging (C2DM) API. But I'm also thinking of building application with the awesome dojox.mobile.

To develop our customer base in the used car experiment, we have created two AdWords campaigns: one for each language, both in the Montreal area. Using AdWords and optimizing the campaigns was very instructive. There are many concepts to master: long tail, auto bid, average CPC, conversion rate, landing page quality score, etc. I know understand why so many people choose to become an AdWords certified partner ;)

Wrapping up 2010, preparing 2011

2010 Summary

2010 was an interesting year for me professionally. Inspired by similar lists online, I present what I did (or can remember at least):

1. Left Compuware and joined my partner Steven at Milstein and Associates inc. end-of-January, to focus 150% of my time on AnotherSocialEconomy (formely known as Twetailer).
2. Adapted the Amazon Flexible Payment Service (FPS) library to the App Engine environment—freely available on github.
3. Refactored the communication layer to be able to send details e-mails to customers, in addition to short ones sent over Twitter and Instant Messaging services.
4. Built the first Web consoles for Golf players and Golf courses staff, based on Dojo and using the freshly delivered REST API—check ezToff.com.
5. Built the first Android application for Golf players using their GPS & Address book to ease the tee-off booking process with AnotherSocialEconomy—freely available on github.
6. Helped preparing pitches to Golf Canada representatives and to Golf staff members and owners.
7. Developed the AnotherSocialEconomy widget, ready to be embedded in participant websites and loading the AnotherSocialEconomy wizard on demand
8. Reviewed the book Google App Engine Java and GWT Application Development.
9. Continued to develop my open-sourced library offering tools for globalizable generic resource bundles (TMX)—on github too.
10. Developed a prototype of a Facebook application.
11. Augmented the AnotherSocialEconomy engine to support the used car dealers: buyers don't buy immediately, but collect car information and offers for a while before committing with one dealer => the engine work flow has been adapted to support this slower path of interaction.
12. Attended presentations to few car dealerlship owners.
13. Attended meetings with various mentors and potential investors.
14. Attended meetings of Montreal NewTech, Android Montreal, Augmented Reality Montreal communities

I’m pretty happy with what I have done so far and am looking forward to doing even more.

New technologies

It was also fun to play around some hot new technologies:

• Ubuntu 10.04 and 10.10
• Android 2.2 and push mechanism on my HTC Desire
• App Engine 1.4.0 and channel api
• Node.js and WebSockets

2011 Goals and Plans

2011 is going to be critical for AnotherSocialEconomy. The application runs and passed usability tests. The focus point is now on the business development!

1. Concentrate on one domain (used car market) and get a significant traffic in the Montreal area.
2. Gather customer feedback (consumer looking for second hand cars and used car dealers), tune the system, and increase traffic. Repeat until 100% satisfaction ;)
3. Once the system is proven by the traffic and testimonies, involve investors and/or partners to 1) expand the business to other areas or 2) to target another domain or 3) both expand geographically and vertically.
4. Develop data mining tools for retailers.
5. Develop domain oriented interfaces for consumers (Web/HTML5 for tablets and PC, native apps for iPhone, Android, BlackBerry).
6. Add more communication channels (like voice messages with Twilio, for example).
7. Offer my services as Software Architect & Developer consultant on designing & developing highly scalable and highly available applications on Google App Engine and mobile applications on Android.

A+, Dom

Reviewed book 'Google App Engine Java and GWT Application Development' is out!

I know that's a pity not to post more regularly! It's just I'm too busy with the developments for AnotherSocialEconomy.com ;)

Here is a little news for Google App Engine developers:

Over the summer, I've been asked to review the draft of the book Google App Engine Java and GWT Application Development. Even with my experience, I learn few techniques, like with the object relationships (chapter 5). A very good book for beginners/intermediates, and still an interesting book for experts.

Enjoy!
A+, Dom

Note: I've no incentive to sell the book, just the pleasure to share a good reference ;)

Securing accounts on the Web

Situation

Few days ago, my partner Steven got his Google account compromised for a short period of time:

• Tweet #1 at 8:52 PM on Sept. 9: Just received 2 calls from friends wondering if I'm being held at a London hotel. FYI, I'm not.
• Tweet #2 at 12:23 AM on Sept. 10: Re: Being held in London. My Google account password was changed by an IP address in Nigeria. I've got it back now but with no Contacts.
• Tweet #3: at 3:04 PM on Sept. 10: Re Stuck in London: I thought I had everything under control last night but needed http://bit.ly/czgYdg Google Security Breach help to fix.

The thieves used his account to send a scam to few of his friends asking for money because he was supposedly blocked in London without resources.

If Steven's password was not very strong, there's no chance it has been discovered after only few attempts. At no time, Google reported that attempts to log into his account were conducted from computers with IP addresses in Liberia! Steven saw the first warning only when he recovered the access!

Encountered risk

The goal of these thieves was limited to getting money as soon as possible. So they reached out few of Steven's contacts, ones he contacts only occasionally, and they asked for a money to be transferred by Western Union. As they kept the control of his account, they would have been able to get the transaction MTCN (money transfer control number) via his inbox. Western Union maintains a page listing the Common Scams.

Others could have decided to change his password, to just spy his incoming message stream (these ones enabled the POP3 and IMAP accesses), to ask for password reset when Steven is not online, and then to steal his identity in many online services.

Because Steven reacted promptly and because his contacts detected the scam, the thieves did not get any benefit from this operation. They are probably trying to get someone else now, maybe someone from his contact list.

How to reduce the exposure

The first protection consists in defining strong passwords. A lot of services offer information about how to produce strong passwords. I would recommend this Microsoft site Strong Passwords | Microsoft Security—I'm confident that they don't provide the online password checker to enhance a grey dictionary ;)

The second protection would be to use a unique and strong password per account. This is probably the most difficult part! I may use probably 20 to 30 online services, some I use regularly, others I use very rarely. There's no way I can remember so many strong passwords...

My solution: Keepass + DropBox

• Keepass is an open source password manager. The tool has been ported on many platforms: Windows, Mac, Linux, iPhone, Android, etc.—Full list on the download page.
• DropBox (link with my referral id ;) is an online file sharing system that, thanks to a program installed on each computer/mobile in your network, maintains in sync the corresponding set of files. DropBox is a nice companion to Keepass as it duplicates your password database transparently, reducing the risk to loose the passwords if the original computer is lost.

The combination of the password generator and Keepass secure edit controls makes the tool especially useful:

• It's easy to generate a strong passwords (remember: 16 characters or plus ;)
• You don't have to remember them as a simple Ctrl+C / Ctrl+V allows to copy securely them in your browser! (the computer clipboard is automatically flushed after few seconds.)

In final, I just have one very strong password (30+ characters) to remember and to change periodically.

Known limitations

Some sites ask users to give a secret answer for a series of predefined questions. If you look at the Apple page below, you'll see that some questions might weaken users more than offering a protection... These days, it's pretty simple to find the responses online!

List of predefined security questions on Apple.com website

Many sites only accepts alphanumerical characters only or don't accept passwords over 20 characters. Oddly enough, most of the bank websites I use prevent too long and too complex passwords! I guess they have other tools to detect intrusions...

List of predefined security questions on Apple.com website

Last minute update

Today, Google announced on its Online Security blog that they will offer a Two-Step Authentication mechanism to log into Google services. This One-Time Password authentication is simpler than distributing a one-time password generator, as Amazon does for example, while providing a still strong security enhancement.

I hope it helps.
A+, Dom

Dilbert: Social Media age

I cannot resist to re-post this strip!




Are such attitudes only due to managers that need to keep a straight control over their reports in order to seamlessly justify their position? I already covered that aspect in my blog post Manager Attitude back in Dec. 2008. I then stated that the situation would be smoother is such managers could fully assume their facilitator role...

These days, I think it's more related to the difference in each person's cognitive age!

In the traditional chronological order, you could become a manager because you knew the company internal mechanics, the management dynamic, and its long term vision. People finger-pointing the Generation Y (see my post Work around the general laziness, for example) have good reasons to ask new employees to first acquire good experiences before giving them rewards and responsibilities.

However, with the emergence of multi-stream platforms, young people have a tremendous possibilities to learn so much in short periods of time and to overpass older ones. My involvement in edu.cyn.in proved this statement many times: kids are much more prolific at producing online content and sharing with their peers than their own teachers!

On the other end of the spectrum, a lot of retired people who don't have to compete again to maintain their social rank, have gain the possibility to become relevant again by being more connected with younger crowds, more socially involved.

This Dilbert strip illustrates a direct confrontation of the socially aged person and the just chronologically aged one!

I hope you like it too.
A+, Dom