I'm trying to get a Custom Security Realm in Glassfish working (i tried 3.0.1 final and 3.1 B33). I read nearly all tutorials about this, but it doesn not work on my System. I'm getting the error 
Login failed: javax.security.auth.login.LoginException: unable to find LoginModule class: de.company.security.utility.CustomLoginModule
when trying to login.
Here is what i did:
I created a little Maven project, which contains the needed Realm class, CustomRealm, and the corresponding LoginModule, CustomLoginModule. 
My pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.mycompany</groupId>
  <artifactId>CustomJDBCRealm</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>Custom JDBCRealm</name>
  <url>http://maven.apache.org</url>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <dependencies>
    <dependency>
        <groupId>org.glassfish.security</groupId>
        <artifactId>security</artifactId>
        <version>3.1-b33</version>
    </dependency>
  </dependencies>
  <build>
      <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.6</source>
                <target>1.6</target>
                <optimise>true</optimise>
                <debug>true</debug>
                <encoding>UTF-8</encoding>
            </configuration>
        </plugin>    
      </plugins>
  </build>
</project>
My Custom Realm class:
package de.company.security.utility;
import com.sun.appserv.security.AppservRealm;
import com.sun.enterprise.security.auth.realm.BadRealmException;
import com.sun.enterprise.security.auth.realm.InvalidOperationException;
import com.sun.enterprise.security.auth.realm.NoSuchRealmException;
import com.sun.enterprise.security.auth.realm.NoSuchUserException;
import java.util.Enumeration;
import java.util.Properties;
import java.util.Vector;
/**
 *
 * @author ifischer
 */
public class CustomRealm extends AppservRealm
{
    Vector<String> groups = new Vector<String>();
    private String jaasCtxName;
    private String startWith;
    @Override
    public void init(Properties properties)
    throws BadRealmException, NoSuchRealmException {
        jaasCtxName = properties.getProperty("jaas-context", "customRealm");
        startWith = properties.getProperty("startWith", "z");
        groups.add("dummy");
    }
    @Override
    public String getAuthType()
    {
        return "Custom Realm";
    }
    public String[] authenticate(String username, char[] password) {
        // if (isValidLogin(username, password))
        return (String[]) groups.toArray();
    }
    @Override
    public Enumeration getGroupNames(String username) throws InvalidOperationException, NoSuchUserException {
        return groups.elements();
    }
    @Override 
    public String getJAASContext()
    {
        return jaasCtxName;
    }
    public String getStartWith()
    {
        return startWith;
    }
}
My LoginModule class:
/*
 * Copyright (c) 2010 ProfitBricks GmbH. All Rights Reserved.
 */
package de.company.security.utility;
import com.sun.appserv.security.AppservPasswordLoginModule;
import com.sun.enterprise.security.auth.login.common.LoginException;
import java.util.Set;
import org.glassfish.security.common.PrincipalImpl;
/**
 *
 * @author ifischer
 */
public class CustomLoginModule extends AppservPasswordLoginModule
{
    @Override
    protected void authenticateUser() throws LoginException
    {
        _logger.info("CustomRealm : authenticateUser for " +  _username);
        final CustomRealm realm = (CustomRealm)_currentRealm;
        if ( (_username == null) || (_username.length() == 0) || !_username.startsWith(realm.getStartWith())) 
            throw new LoginException("Invalid credentials");
        String[] grpList = realm.authenticate(_username, getPasswordChar()); 
        if (grpList == null) { 
            throw new LoginException("User not in groups");
        }
        _logger.info("CustomRealm : authenticateUser for " +  _username);
        Set principals = _subject.getPrincipals();
        principals.add(new PrincipalImpl(_username));
        this.commitUserAuthentication(grpList);
    }
}
I compiled this Maven project and copyied the resulting JAR-file to the Glassfish/lib directory. Then i added the Security Realm "customRealm" to my Glassfish with asadmin:
asadmin create-auth-realm 
  --classname de.company.security.utility.CustomRealm 
  --property jaas-context=customRealm:startWith=a customRealm
I even referenced the LoginModule class for the JAAS context of my Custom Realm, therefore i inserted this into the login.conf of my domain:
customRealm {
  de.company.security.utility.CustomLoginModule required;
};
Although this LoginModule SHOULD BE on the Glassfish classpath, as it's classfiled is packaged in the JAR that i put into the Glassfish/lib-dir, it cannot be found when i try to login. For login, i build a really simple JSF-project, which calls the HttpServletRequest-login-method of Servlet 3.0. 
When trying to login i'm getting the following Exception:
2010-12-24T14:41:31.613+0100|WARNING|glassfish3.0.1|
javax.enterprise.system.container.web.com.sun.web.security|_ThreadID=25;
_ThreadName=Thread-1;|Web login failed: Login failed: 
javax.security.auth.login.LoginException: unable to find LoginModule class:
de.company.security.utility.CustomLoginModule
Anybody got an idea what i can do that Glassfish loads the LoginModule-class?