I'm trying to install bash as the default shell on a ARM Linux running on an embedded device (Synology DS212+ NAS). But there's something really wrong, and I can't figure out what it is.

Symptoms:

1) Root has /bin/bash as default shell, and can log in normally via SSH:

$ grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash

$ ssh root@NAS
root@NAS's password:
Last login: Sun Dec 16 14:06:56 2012 from desktop
# 

2) joeuser has /bin/bash as default shell, and receives "Permission denied" when trying to log in via SSH:

$ grep joeuser /etc/passwd
joeuser:x:1029:100:Joe User:/home/joeuser:/bin/bash

$ ssh joeuser@localhost
joeuser@NAS's password:
Last login: Sun Dec 16 14:07:22 2012 from desktop
Permission denied, please try again.
Connection to localhost closed.

3) changing joeuser's shell back to /bin/sh:

$ grep joeuser /etc/passwd
joeuser:x:1029:100:Joe User:/home/joeuser:/bin/sh

$ ssh joeuser@localhost
Last login: Sun Dec 16 15:50:52 2012 from localhost
$ 

To make things even more strange, I can log in as joeuser using /bin/bash using the serial console (!). Also a su - joeuser as root works fine, so the bash binary itself is working fine.

In an act of despair, I changed joeuser's uid to 0 on /etc/passwd, but also didn't work, so it doesn't seem to be anything permission related.

Seems that bash is doing some extra checking that sshd didn't like, and blocking the connections for non-root users. Maybe some sort of sanity checking - or terminal emulation - that is triggering the SIGCHLD, but only when called via ssh.

I already went through every single item on sshd_config, and also put SSHD in debug mode, but didn't find anything strange. Here's my /etc/ssh/sshd_config:

LogLevel DEBUG
LoginGraceTime 2m
PermitRootLogin yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      %h/.ssh/authorized_keys
ChallengeResponseAuthentication no
UsePAM yes
AllowTcpForwarding no
ChrootDirectory none
Subsystem       sftp    internal-sftp -f DAEMON -u 000

And here's the output from /usr/syno/sbin/sshd -d, showing the failed attempt of joeuser trying to log in, with /bin/bash as the shell:

debug1: Config token is loglevel
debug1: Config token is logingracetime
debug1: Config token is permitrootlogin
debug1: Config token is rsaauthentication
debug1: Config token is pubkeyauthentication
debug1: Config token is authorizedkeysfile
debug1: Config token is challengeresponseauthentication
debug1: Config token is usepam
debug1: Config token is allowtcpforwarding
debug1: Config token is chrootdirectory
debug1: Config token is subsystem
debug1: HPN Buffer Size: 87380
debug1: sshd version OpenSSH_5.8p1-hpn13v11
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/syno/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_adj from 0 to -17
debug1: Bind to port 22 on ::.
debug1: Server TCP RWIN socket size: 87380
debug1: HPN Buffer Size: 87380
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
debug1: Server TCP RWIN socket size: 87380
debug1: HPN Buffer Size: 87380
Server listening on 0.0.0.0 port 22.

debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 6 out 6 newsock 6 pipe -1 sock 9
debug1: inetd sockets after dupping: 4, 4
Connection from 127.0.0.1 port 52212
debug1: HPN Disabled: 0, HPN Buffer Size: 87380
debug1: Client protocol version 2.0; client software version OpenSSH_5.8p1-hpn13v11
SSH: Server;Ltype: Version;Remote: 127.0.0.1-52212;Protocol: 2.0;Client: OpenSSH_5.8p1-hpn13v11
debug1: match: OpenSSH_5.8p1-hpn13v11 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1-hpn13v11
debug1: permanently_set_uid: 1024/100
debug1: MYFLAG IS 1
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: AUTH STATE IS 0
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: client->server aes128-ctr hmac-md5 none
SSH: Server;Ltype: Kex;Remote: 127.0.0.1-52212;Enc: aes128-ctr;MAC: hmac-md5;Comp: none
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: expecting SSH2_MSG_KEX_ECDH_INIT
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user joeuser service ssh-connection method none
SSH: Server;Ltype: Authname;Remote: 127.0.0.1-52212;Name: joeuser
debug1: attempt 0 failures 0
debug1: Config token is loglevel
debug1: Config token is logingracetime
debug1: Config token is permitrootlogin
debug1: Config token is rsaauthentication
debug1: Config token is pubkeyauthentication
debug1: Config token is authorizedkeysfile
debug1: Config token is challengeresponseauthentication
debug1: Config token is usepam
debug1: Config token is allowtcpforwarding
debug1: Config token is chrootdirectory
debug1: Config token is subsystem
debug1: PAM: initializing for "joeuser"
debug1: PAM: setting PAM_RHOST to "localhost"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user joeuser service ssh-connection method password
debug1: attempt 1 failures 0
debug1: do_pam_account: called
Accepted password for joeuser from 127.0.0.1 port 52212 ssh2
debug1: monitor_child_preauth: joeuser has been authenticated by privileged process
debug1: PAM: establishing credentials
User child is on pid 9129
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype [email protected] want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.

debug1: Received SIGCHLD.
debug1: session_by_pid: pid 9130
debug1: session_exit_message: session 0 channel 0 pid 9130
debug1: session_exit_message: release channel 0
debug1: session_by_tty: session 0 tty /dev/pts/1
debug1: session_pty_cleanup: session 0 release /dev/pts/1
Received disconnect from 127.0.0.1: 11: disconnected by user
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials

Here you have the full output of sshd -dd, together with ssh -vv.

Bash:

# bash --version
GNU bash, version 3.2.49(1)-release (arm-none-linux-gnueabi)
Copyright (C) 2007 Free Software Foundation, Inc.

The bash binary was cross compiled from source. I also tried using a pre-compiled binary from the Optware distribution, but had the exact same problem. I checked for missing shared libraries using objdump -x, but they're all there.

Any ideas what could be causing this "Permission denied, please try again."? I'm almost diving in the bash source code to investigate, but trying to avoid hours chasing something that may be silly.