End of the year 2010 reflection

A lot had happened this year,  and so many important decisions were made. Leaving previous companies and accepting a new position at Intel Indonesia were the key decisions. Engineering tasks have always been my passion. Hands-on design and coding, tackling clients' problem, out of the box thinking on problem solving had always been my day to day tasks.

For the last 8 months, I have moved on to Intel Indonesia doing Intel Embedded sales. A new role but it still requires deep engineering experience to help customers out on making technical decisions. But hey.. I can't just get rid of your engineering mind. I just can't get rid of the itch when I see some technical issues that require hands on approach. So, I hope I can dump some that I have learnt next year. Apology to the comment poster if I did not reply your comment posts immediately because I have not activate Blogspot comment posts' email notification yet.

Using Zimbra LDAP as Spring-security backend and handling Linux/BSD MD5 encrypted password

LDAP has been one of the most common way of managing user credentials for most company. Email and other services may use LDAP to authenticate users. I have been using with Spring Security for quite sometime however recently I have a very challenging task of integrating Zimbra LDAP as Spring Security backend.

Zimbra LDAP does not allow user to connect and authenticate directly (no binding authentication). The preferred way of authenticating user is by comparing password (see PasswordComparisonAuthenticator).

My scenario is simple:

1. First, Use initialDirContextFactory to connect to LDAP using the supplied admin/manager credential
2. Use an LDAPAuthenticationProvider to do username password comparison and then to supply a list of granted authorities to each authenticated user

However, in my case this scenario is becoming complicated because:

1. First, Zimbra LDAP stores password as a BSD/Linux MD5 Crypt format. i.e. {crypt}$1$[salt]$[encrypted_passwd] because it was generated from the entries of /etc/password from our old mail system
2. Second, the crypt compatible implementation of Spring Security PasswordEncoder does not exist and the default PasswordComparisonAuthenticator does not support salt
3. Third, standard Zimbra LDAP does not store any group information. And what I need is just to grant a "ROLE_USER" to each authenticated user.

So here is what I have done:

1. Implement a PasswordEncoder (CryptPasswordEncoder) that is compatible with Linux/BSD Crypt
2. Implement a PasswordComparisonAuthenticator (CryptPasswordAuthenticator) that is capable of fetching salt from LDAP and then use that salt to encode the password for further PasswordComparison
3. Implement a SimpleLdapAuthoritiesPopulator to fill in the desired granted authorities I wish to give for any authenticated user

Here is my springsecurityContext.xml configuration for that:
First we need to define the connection to LDAP which is capable of listing all uid.

<bean id="initialDirContextFactory"   
    class="org.springframework.security.ldap.DefaultSpringSecurityContextSource">
        <constructor-arg value="ldap://your_zimbra_server/ou=people,dc=your_domain,dc=com" />
        <property name="userDn">
            <value>uid=zimbra,cn=admins,cn=zimbra</value>
        </property>
        <property name="password">
            <value>your_zimbra_ldap_password</value>
        </property>
    </bean> 

    <bean id="passwordEncoder" class="jbs.CryptPasswordEncoder" />
    <bean id="defaultga" class="org.springframework.security.core.authority.GrantedAuthorityImpl">
        <constructor-arg  value="ROLE_USER"/>
    </bean>
    <bean id="authenticationProvider" class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">
        <constructor-arg>
            <bean class="jbs.CryptPasswordComparisonAuthenticator">
                <constructor-arg ref="initialDirContextFactory" />
                <property name="passwordAttributeName" value="userPassword"/>
                <property name="userDnPatterns" value="uid={0}"/>
                <property name="passwordEncoder" ref="passwordEncoder"/>
            </bean>
        </constructor-arg>
        <constructor-arg>
            <bean class="jbs.SimpleLdapAuthoritiesPopulator">
                <property name="defaultGrantedAuthorities">
                    <list>
                        <ref bean="defaultga"/>
                    </list>
                </property>
            </bean>
        </constructor-arg>
    </bean>

   <security:authentication-manager>
        <security:authentication-provider ref="authenticationProvider"/>
   </security:authentication-manager>

Here is my implementation of CryptPasswordEncoder:

package jbs;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.security.authentication.encoding.PasswordEncoder;

/**
 *
 * @author joko
 */
public class CryptPasswordEncoder implements PasswordEncoder {

    private static final String CRYPT_PREFIX = "{crypt}";
    private Log log = LogFactory.getLog(this.getClass().getName());

    @Override
    public String encodePassword(String rawPass, Object salt) throws DataAccessException {
        String ret = "";
        try {
            ret = CRYPT_PREFIX + MD5Crypt.crypt(rawPass, (salt!=null)? salt.toString(): "");            
        } catch (Exception ex) {
            log.warn(ex.getMessage());
        }
        return ret;
    }

    @Override
    public boolean isPasswordValid(String encPass, String rawPass, Object salt) throws DataAccessException {
        boolean ret = false;
        if (encPass.startsWith(CRYPT_PREFIX)) {
            //strip header
            String saltedhash = encPass.substring(CRYPT_PREFIX.length() + 3);
            log.debug("Salted hash is: " + saltedhash);
            // remove salt
            String[] arr = saltedhash.split("\\$");
            if (arr.length > 1) {
                ret = encodePassword(rawPass, salt).endsWith(arr[1]);
            } else {
                ret = encodePassword(rawPass, salt).endsWith(saltedhash);
            }
        }
        return ret;
    }
}

For class MD5Crypt and its dependencies, please have a look at Applied Research Laboratories GIT Server

My CryptPasswordAuthenticator.java is basically very similar to the original PasswordAuthenticator.java however I fetch the salt from the userPassword field of LDAP user entry. Then I use the salt to generate the encrypted password for further LDAP comparison.

package jbs;

import java.lang.reflect.Method;
import javax.naming.NamingEnumeration;
import javax.naming.directory.Attribute;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.ldap.NameNotFoundException;
import org.springframework.ldap.NamingException;
import org.springframework.ldap.core.DirContextOperations;
import org.springframework.ldap.core.support.BaseLdapPathContextSource;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.encoding.PasswordEncoder;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.security.ldap.LdapUtils;
import org.springframework.security.ldap.SpringSecurityLdapTemplate;
import org.springframework.security.ldap.authentication.AbstractLdapAuthenticator;
import org.springframework.util.Assert;

/**
 *
 * @author joko
 */
public class CryptPasswordComparisonAuthenticator extends AbstractLdapAuthenticator {

    private static final Log logger = LogFactory.getLog(CryptPasswordComparisonAuthenticator.class);
    //~ Instance fields ================================================================================================
    private PasswordEncoder passwordEncoder = new CryptPasswordEncoder();
    private String passwordAttributeName = "userPassword";

    //~ Constructors ===================================================================================================
    public CryptPasswordComparisonAuthenticator(BaseLdapPathContextSource contextSource) {
        super(contextSource);
    }

    @Override
    public DirContextOperations authenticate(final Authentication authentication) {
        DirContextOperations user = null;
        String username = authentication.getName();
        String password = (String) authentication.getCredentials();

        SpringSecurityLdapTemplate ldapTemplate = new SpringSecurityLdapTemplate(getContextSource());

        for (String userDn : getUserDns(username)) {
            try {
                user = ldapTemplate.retrieveEntry(userDn, getUserAttributes());
            } catch (NameNotFoundException ignore) {
            }
            if (user != null) {
                break;
            }
        }

        if (user == null && getUserSearch() != null) {
            user = getUserSearch().searchForUser(username);
        }

        if (user == null) {
            throw new UsernameNotFoundException("User not found: " + username, username);
        }


        if (logger.isDebugEnabled()) {
            logger.debug("Performing LDAP compare of password attribute '" + passwordAttributeName + "' for user '" + user.getDn() + "'");
        }
        String storedPassword = new String((byte[]) (user.getObjectAttribute(passwordAttributeName)));
        // get the salt
        String[] arr = storedPassword.split("\\$");        
        String encodedPassword = passwordEncoder.encodePassword(password, arr.length > 1 ? arr[2] : null);
        byte[] passwordBytes = LdapUtils.getUtf8Bytes(encodedPassword);


        logger.debug(encodedPassword + ":" + storedPassword);
        if (!ldapTemplate.compare(user.getDn().toString(), passwordAttributeName, passwordBytes)) {
            throw new BadCredentialsException(messages.getMessage("PasswordComparisonAuthenticator.badCredentials",
                    "Bad credentials"));
        }

        return user;
    }

    public void setPasswordAttributeName(String passwordAttribute) {
        Assert.hasLength(passwordAttribute, "passwordAttributeName must not be empty or null");
        this.passwordAttributeName = passwordAttribute;
    }

    public void setPasswordEncoder(PasswordEncoder passwordEncoder) {
        Assert.notNull(passwordEncoder, "passwordEncoder must not be null.");
        this.passwordEncoder = passwordEncoder;
    }
}

The final Java class that I write which is SimpleLdapAuthoritiesPopulator.java is as following:

package jbs;

import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.directory.Attribute;
import org.apache.commons.logging.LogFactory;
import org.springframework.ldap.core.DirContextOperations;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.ldap.userdetails.LdapAuthoritiesPopulator;

/**
 *
 * @author joko
 */
public class SimpleLdapAuthoritiesPopulator implements LdapAuthoritiesPopulator{

    private List<GrantedAuthority> defaultGrantedAuthorities = new ArrayList<GrantedAuthority>();

    @Override
    public Collection<GrantedAuthority> getGrantedAuthorities(DirContextOperations userData, String username) {
        NamingEnumeration<? extends Attribute> nes = userData.getAttributes().getAll();
        try {
            while (nes.hasMore()) {
                Attribute a = nes.next();
                a.toString();
            }
        } catch (NamingException ex) {
            LogFactory.getLog(SimpleLdapAuthoritiesPopulator.class.getName()).warn(ex.getMessage());
        }
        return getDefaultGrantedAuthorities();
    }

    /**
     * @return the defaultGrantedAuthorities
     */
    public List<GrantedAuthority> getDefaultGrantedAuthorities() {
        return defaultGrantedAuthorities;
    }

    /**
     * @param defaultGrantedAuthorities the defaultGrantedAuthorities to set
     */
    public void setDefaultGrantedAuthorities(List<GrantedAuthority> defaultGrantedAuthorities) {
        this.defaultGrantedAuthorities = defaultGrantedAuthorities;
    }

}

I hope this would help anyone who are facing the same trouble as I did.

Using javax.xml.soap to access OTRS SOAP service without WSDL

For most of us who are frequently using WSDL to generate stub classes for accessing webservices, dealing with SOAP services without WSDL is painful. OTRS (Open source Ticket Request System) is one of them. One attempt by OpenNMS team has been quite successful on integrating OTRS by writing its own webservice as an OTRS plugin. However, this is not satisfactory in my opinion. We need a better way of doing it.

I have tried to find out the pattern of the SOAP messages being exchanged by the rpc-example.pl with my local installation of OTRS. Here is the process of the investigation and this attempt has successfully execute RPC on the OTRS and I have been able to retrieve the result.

First, I create a sample perl script to make a simple request to retrieve Ticket by TicketID. Here is the script:

#!/usr/bin/perl -w
use SOAP::Lite ( 'autodispatch', proxy => 'http://localhost/otrs/rpc.pl', trace => 'all');
my $User = 'some_user';
my $Pw   = 'some_pass';

my $RPC = Core->new();
my %Ticket = $RPC->Dispatch($User,$Pw,'TicketObject','TicketGet', TicketID=>1);
print "$Ticket{TicketNumber}\n";

Note: I set 'trace' property to 'all' to enable printing out every possible debugging message for SOAP::Lite API

Observing the output of the debug message, I found the SOAP request has the following pattern:

<SOAP:Envelope xmlns ... dst..dst>
    <soap:Body>
          <Dispatch xmlns="/Core">
                  <Username xsi:type="xsd:string">some_user</Username>
                  <Password xsi:type="xsd:string">some_pass</Password>
                  <Object xsi:type="xsd:string">TicketObject or other object </Object>
                  <Method xsi:type="xsd:string">TicketGet or other method</Method>
                  <Param1_Name xsi:type="xsd:string">fill in the param name</Param1_Name>
                  <Param1_Value xsi:type="xsd:string/int">fill in the param value</Param1_Value>
...
... other params
          </Dispatch>
    </soap:Body>
</SOAP:Envelope>

Note: I have modified the element name to make it easier to understand what the content supposed to be

Having looked at the pattern, I then wrote a sample Java code to access it. Here is my attempt:

import java.net.URL;
import javax.xml.namespace.QName;
import javax.xml.soap.MessageFactory;
import javax.xml.soap.SOAPBody;
import javax.xml.soap.SOAPBodyElement;
import javax.xml.soap.SOAPConnection;
import javax.xml.soap.SOAPConnectionFactory;
import javax.xml.soap.SOAPEnvelope;
import javax.xml.soap.SOAPMessage;
import javax.xml.soap.SOAPPart;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;


/**
 *
 * @author masjoko
 */
public class Main {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws Exception {
        // Create the connection
 SOAPConnectionFactory scf = SOAPConnectionFactory.newInstance();
 SOAPConnection conn = scf.createConnection();

 // Create message
 MessageFactory mf = MessageFactory.newInstance();
 SOAPMessage msg = mf.createMessage();

 // Object for message parts
 SOAPPart sp = msg.getSOAPPart();

 SOAPEnvelope env = sp.getEnvelope();
 env.addNamespaceDeclaration("xsd","http://www.w3.org/2001/XMLSchema");
 env.addNamespaceDeclaration("xsi","http://www.w3.org/2001/XMLSchema-instance");
 env.addNamespaceDeclaration("enc","http://schemas.xmlsoap.org/soap/encoding/");
 env.addNamespaceDeclaration("env","http://schemas.xmlsoap.org/soap/envelop/");
 env.setEncodingStyle("http://schemas.xmlsoap.org/soap/encoding/");

        SOAPBody body = env.getBody();
        SOAPBodyElement dispatch = body.addBodyElement(new QName("/Core","Dispatch"));
        dispatch.addChildElement("Username").addTextNode("some_user").setAttribute("xsi:type", "xsd:string");
        dispatch.addChildElement("Password").addTextNode("some_pass").setAttribute("xsi:type", "xsd:string");
        dispatch.addChildElement("Object").addTextNode("TicketObject").setAttribute("xsi:type", "xsd:string");
        dispatch.addChildElement("Method").addTextNode("TicketGet").setAttribute("xsi:type", "xsd:string");
        dispatch.addChildElement("Param1_Name").addTextNode("TicketID").setAttribute("xsi:type", "xsd:string");
        dispatch.addChildElement("Param1_Value").addTextNode("1").setAttribute("xsi:type", "xsd:int");
        URL url = new URL("http://localhost/otrs/rpc.pl");
        SOAPMessage resp = conn.call(msg, url);
        Document doc = resp.getSOAPPart().getEnvelope().getBody().extractContentAsDocument();
        Element el = doc.getDocumentElement();
        NodeList nl = el.getChildNodes();
        for (int i = 0 ; i<(nl.getLength()/2); i++)
        {
            System.out.println(nl.item(i*2).getTextContent().trim()+":"+nl.item(i*2+1).getTextContent().trim());
        }
    }
}

The following output was shown:

Age:91060917
PriorityID:3
ServiceID:
TicketFreeText11:
TicketFreeTime4:
TicketFreeTime1:
TicketFreeText6:
StateID:6
TicketFreeTime5:
EscalationTime:0
TicketFreeTime6:
TicketFreeKey9:
OwnerID:1
Owner:root@localhost
TicketFreeText7:
TicketFreeKey11:
Created:2007-03-08 16:26:39
TicketFreeText4:
QueueID:4
TicketFreeText2:
TicketFreeKey6:
TicketID:1
TicketFreeKey5:
TicketFreeText12:
UnlockTimeout:1264399466
EscalationResponseTime:0
TicketFreeTime3:
TicketFreeText3:
CustomerUserID:
TicketFreeText8:
TicketFreeText9:
Type:default
TicketFreeKey7:
Responsible:root@localhost
TicketFreeText10:
ResponsibleID:1
TicketFreeKey16:
TicketFreeKey3:
RealTillTimeNotUsed:1264485840
GroupID:1
TicketFreeKey13:
CustomerID:
TicketFreeKey1:
TypeID:1
Priority:3 normal
TicketFreeKey12:
TicketFreeKey10:
TicketFreeKey8:
UntilTime:78924
TicketFreeText1:
EscalationUpdateTime:0
TicketFreeTime2:
Queue:Misc
TicketFreeText13:
State:pending reminder
Title:Welcome to OTRS!
TicketFreeText5:
TicketFreeText15:
TicketFreeText14:
StateType:pending reminder
EscalationSolutionTime:0
LockID:1
TicketFreeKey2:
TicketNumber:1010001
TicketFreeKey14:
Lock:unlock
CreateTimeUnix:1173345999
TicketFreeKey4:
SLAID:
TicketFreeKey15:
TicketFreeText16:

Now, I believe someone should try to write a general purpose method/utility to call OTRS method using Java. I hope this would help anyone who encountered similar problem.

Accessing WS-Security protected (UsernameToken) WebService using PHP5 Soap

After battling for several hour trying to figure out on how to send WS-Security header using PHP5 Soap API, I finally managed to discover the work of the good people at University of Toronto . Using their example and the PHP code they posted on the website: soap-wsse.php and xmlseclibs.php, I have been able to test my CXF-powered webservice.

I have a web service running at http://localhost:9090/WS with WSDL url of http://localhost:9090/WS?wsdl. Using the following PHP script and following the direction from the University of Toronto (putting the two PHP scripts above in the same folder as my script), I have been able to call my webservice correctly.

<?php
require('soap-wsse.php');

class mySoap extends SoapClient {
   function __doRequest($request, $location, $saction, $version) {
    $doc = new DOMDocument('1.0');
    $doc->loadXML($request);
    $objWSSE = new WSSESoap($doc);
    $objWSSE->addUserToken("admin", "admin", false);
    return parent::__doRequest($objWSSE->saveXML(), $location, $saction, $version);
   }
}

$wsdl = 'http://localhost:9090/WS?wsdl';
$sClient = new mySoap($wsdl, array('trace'=>1));
$wrapper->word = new SoapVar("Echo testing", XSD_STRING);
$result = $sClient->echo($wrapper);
print_r($result->return);
?>

The resulting print out on the web is "Echo Testing". I hope this would help anyone that encountered similar problem.

Mono WebClient Basic Authorization pre-authenticate problems

For the last few week I have been trying to set Mono WebClient to send (.. yeah.. simple... just send the fricking..) Basic Authorization header. Apparently, somebody is just too smart to decide not sending the header on the first request for security reason (sending user credential).

Well, the problem is that Acegi/Spring security does not necessarily send 401 header since authentication can be done using Authorization header as well as form. I am using code that works perfectly on Microsoft .NET 2.0/3.5. I have not been able to get it to work on Mono on FC11.

I wish somebody really looked at the problem and have it fixed soon otherwise I have to use Java and Apache Commons HTTP.