I just recently set up a mixture of RHEL 6.1 Rackspace cloud hosts and RHEL 6.2 dedicated hosts using Rackconnect. I am experiencing intermittent Redis timeouts from within our Rails 3.2.8 app with Redis 2.4.16 running on the RHEL 6.2 dedicated hosts. There is no network latency or packet loss. Also there are no errors on any interfaces on our cloud or dedicated servers or on the managed firewall from Rackspace.

When Redis timesout, there is nothing logged within redis even though it is set up to do debug logging. The only error we receive is from Airbrake saying there was a Redis timeout.

Network topology:

RHEL 6.1 cloud hosts  <--> Alert logic IDS <--> Cisco ASA 5510 <--> RHEL 6.2 dedicated hosts
(web nodes)                            (two way NAT)             (db hosts running redis)

Ping from db host to web host:

64 bytes from 10.181.230.180: icmp_seq=998 ttl=64 time=0.520 ms
64 bytes from 10.181.230.180: icmp_seq=999 ttl=64 time=0.579 ms
64 bytes from 10.181.230.180: icmp_seq=1000 ttl=64 time=0.482 ms

--- web1.xxxxxx.com ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 999007ms
rtt min/avg/max/mdev = 0.359/0.535/5.684/0.200 ms

Ping from web host to db host:

64 bytes from 192.168.100.26: icmp_seq=998 ttl=64 time=0.544 ms
64 bytes from 192.168.100.26: icmp_seq=999 ttl=64 time=0.452 ms
64 bytes from 192.168.100.26: icmp_seq=1000 ttl=64 time=0.529 ms

--- data1.xxxxxx.com ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 999017ms
rtt min/avg/max/mdev = 0.358/0.499/6.120/0.201 ms

Redis config:

daemonize yes
pidfile /var/run/redis/6379/redis_6379.pid

port 6379

timeout 0

loglevel debug

logfile /var/lib/redis/log

syslog-enabled yes

syslog-ident redis-6379

syslog-facility local0

databases 16

save 900 1
save 300 10
save 60 10000

rdbcompression yes

dbfilename dump-6379.rdb

dir /var/lib/redis

maxclients 10000

maxmemory-policy volatile-lru

maxmemory-samples 3

appendfilename appendonly-6379.aof

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

slowlog-log-slower-than 10000

slowlog-max-len 1024

vm-enabled no

vm-swap-file /tmp/redis.swap

vm-max-memory 0

vm-page-size 32

vm-pages 134217728

vm-max-threads 4

hash-max-zipmap-entries 512
hash-max-zipmap-value 64

list-max-ziplist-entries 512
list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128
zset-max-ziplist-value 64

activerehashing yes

Redis-cli info:

redis-cli info
redis_version:2.4.16
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:4174
uptime_in_seconds:79346
uptime_in_days:0
lru_clock:1064644
used_cpu_sys:13.08
used_cpu_user:19.81
used_cpu_sys_children:1.56
used_cpu_user_children:7.69
connected_clients:167
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:6
used_memory:15060312
used_memory_human:14.36M
used_memory_rss:22061056
used_memory_peak:15265928
used_memory_peak_human:14.56M
mem_fragmentation_ratio:1.46
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:166
bgsave_in_progress:0
last_save_time:1352823542
bgrewriteaof_in_progress:0
total_connections_received:286
total_commands_processed:507254
expired_keys:0
evicted_keys:0
keyspace_hits:1509
keyspace_misses:65167
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:690
vm_enabled:0
role:master
db0:keys=6,expires=0

edit 1: add redis-cli info output