solved: puppet master REST API returns 403 when running under passenger works when master runs from command line
- by Anadi Misra
I am using the standard auth.conf provided in puppet install for the puppet master which is running through passenger under Nginx. However for most of the catalog, files and certitifcate request I get a 403 response.
### Authenticated paths - these apply only when the client
### has a valid certificate and is thus authenticated
# allow nodes to retrieve their own catalog
path ~ ^/catalog/([^/]+)$
method find
allow $1
# allow nodes to retrieve their own node definition
path ~ ^/node/([^/]+)$
method find
allow $1
# allow all nodes to access the certificates services
path ~ ^/certificate_revocation_list/ca
method find
allow *
# allow all nodes to store their reports
path /report
method save
allow *
# unconditionally allow access to all file services
# which means in practice that fileserver.conf will
# still be used
path /file
allow *
### Unauthenticated ACL, for clients for which the current master doesn't
### have a valid certificate; we allow authenticated users, too, because
### there isn't a great harm in letting that request through.
# allow access to the master CA
path /certificate/ca
auth any
method find
allow *
path /certificate/
auth any
method find
allow *
path /certificate_request
auth any
method find, save
allow *
path /facts
auth any
method find, search
allow *
# this one is not stricly necessary, but it has the merit
# of showing the default policy, which is deny everything else
path /
auth any
Puppet master however does not seems to be following this as I get this error on client
[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose --server bangvmpllda02.XXXXX.com
[sudo] password for amisr1:
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /certificate_revocation_list/ca [find] at :110
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [search] at :110
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110 Could not retrieve file metadata for puppet://devops.XXXXX.com/plugins: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /file_metadata/plugins [find] at :110
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /catalog/blramisr195602.XXXXX.com [find] at :110
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: XX.XXX.XX.XX(XX.XXX.XX.XX) access to /report/blramisr195602.XXXXX.com [save] at :110
and the server logs show
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/certificate_revocation_list/ca? HTTP/1.1" 403 102 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadatas/plugins?links=manage&recurse=true&&ignore=---+%0A++-+%22.svn%22%0A++-+CVS%0A++-+%22.git%22&checksum_type=md5 HTTP/1.1" 403 95 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:52 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "POST /production/catalog/blramisr195602.XXXXX.com HTTP/1.1" 403 106 "-" "Ruby"
XX.XXX.XX.XX - - [10/Dec/2012:14:46:53 +0530] "PUT /production/report/blramisr195602.XXXXX.com HTTP/1.1" 403 105 "-" "Ruby"
thefile server conf file is as follows (and goin by what they say on puppet site, It is better to regulate access in auth.conf for reaching file server and then allow file server to server all)
[files]
path /apps/puppet/files
allow *
[private]
path /apps/puppet/private/%H
allow *
[modules]
allow *
I am using server and client version 3
Nginx has been compiled using the following options
nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/apps/nginx --conf-path=/apps/nginx/nginx.conf --pid-path=/apps/nginx/run/nginx.pid --error-log-path=/apps/nginx/logs/error.log --http-log-path=/apps/nginx/logs/access.log --with-http_ssl_module --with-http_gzip_static_module --add-module=/usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/nginx --add-module=/apps/Downloads/nginx/nginx-auth-ldap-master/
and the standard nginx puppet master conf
server {
ssl on;
listen 8140 ssl;
server_name _;
passenger_enabled on;
passenger_set_cgi_param HTTP_X_CLIENT_DN $ssl_client_s_dn;
passenger_set_cgi_param HTTP_X_CLIENT_VERIFY $ssl_client_verify;
passenger_min_instances 5;
access_log logs/puppet_access.log;
error_log logs/puppet_error.log;
root /apps/nginx/html/rack/public;
ssl_certificate /var/lib/puppet/ssl/certs/bangvmpllda02.XXXXXX.com.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/bangvmpllda02.XXXXXX.com.pem;
ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;
ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem;
ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_prefer_server_ciphers on;
ssl_verify_client optional;
ssl_verify_depth 1;
ssl_session_cache shared:SSL:128m;
ssl_session_timeout 5m;
}
Puppet is picking up the correct settings from the files mentioned because config print command points to /etc/puppet
[amisr1@bangvmpllDA02 puppet]$ sudo puppet config print | grep conf
async_storeconfigs = false
authconfig = /etc/puppet/namespaceauth.conf
autosign = /etc/puppet/autosign.conf
catalog_cache_terminus = store_configs
confdir = /etc/puppet
config = /etc/puppet/puppet.conf
config_file_name = puppet.conf
config_version = ""
configprint = all
configtimeout = 120
dblocation = /var/lib/puppet/state/clientconfigs.sqlite3
deviceconfig = /etc/puppet/device.conf
fileserverconfig = /etc/puppet/fileserver.conf
genconfig = false
hiera_config = /etc/puppet/hiera.yaml
localconfig = /var/lib/puppet/state/localconfig
name = config
rest_authconfig = /etc/puppet/auth.conf
storeconfigs = true
storeconfigs_backend = puppetdb
tagmap = /etc/puppet/tagmail.conf
thin_storeconfigs = false
I checked the firewall rules on this VM; 80, 443, 8140, 3000 are allowed. Do I still have to tweak any specifics to auth.conf for getting this to work?
Update
I added verbose logging to the puppet master and restarted nginx; here's the additional info I see in logs
Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Could not resolve 10.209.47.31: no name for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 access[/] (info): defaulting to no access for 10.209.47.31
Mon Dec 10 18:19:15 +0530 2012 Puppet (warning): Denying access: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
Mon Dec 10 18:19:15 +0530 2012 Puppet (err): Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :111
10.209.47.31 - - [10/Dec/2012:18:19:15 +0530] "GET /production/file_metadata/plugins? HTTP/1.1" 403 93 "-" "Ruby"
On the agent machine facter fqdn and hostname both return a fully qualified host name
[amisr1@blramisr195602 ~]$ sudo facter fqdn
blramisr195602.XXXXXXX.com
I then updated the agent configuration to add
dns_alt_names = 10.209.47.31
cleaned all certificates on master and agent and regenerated the certificates and signed them on master using the option --allow-dns-alt-names
[amisr1@bangvmpllDA02 ~]$ sudo puppet cert sign blramisr195602.XXXXXX.com
Error: CSR 'blramisr195602.XXXXXX.com' contains subject alternative names (DNS:10.209.47.31, DNS:blramisr195602.XXXXXX.com), which are
disallowed. Use `puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com` to sign this request.
[amisr1@bangvmpllDA02 ~]$ sudo puppet cert --allow-dns-alt-names sign blramisr195602.XXXXXX.com
Signed certificate request for blramisr195602.XXXXXX.com
Removing file Puppet::SSL::CertificateRequest blramisr195602.XXXXXX.com at '/var/lib/puppet/ssl/ca/requests/blramisr195602.XXXXXX.com.pem'
however, that doesn't help either; I get same errors as before. Not sure why in the logs it shows comparing access rules by IP and not hostname. Is there any Nginx configuration to change this behavior?
