Both RewriteRule's works fine, except when used together.

1.Remove all queries except query ?callback=.*:

# /api?callback=foo       has no rewrite
# /whatever?whatever=foo  has 301 redirect  /whatever
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^?#\ ]*)\?[^\ ]*\ HTTP/ [NC]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} !/api(/.*)?\?callback=.*
RewriteRule .*$ %{REQUEST_URI}? [R=301,L]

2.Rewrite index.php queries api and url=$1:

# /api           returns data  index.php?api&url=
# /api/whatever  returns data  index.php?api&url=whatever
RewriteRule ^api(?:/([^/]*))?$ index.php?api&url=$1 [QSA,L]
RewriteRule ^([^.]*)$ index.php?url=$1 [QSA,L]

Any valid combination to this RewriteRule's on keeping its functionality?

This combination will return Server Error 404 to /api/?callback=foo:

# Remove all queries except query "callback"
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^?#\ ]*)\?[^\ ]*\ HTTP/ [NC]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} !/api(/.*)?\?callback=.*
RewriteRule .*$ %{REQUEST_URI}? [R=301,L]

# Rewrite index.php queries
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} !/api(/.*)?\?callback=.*
# Server Error 404 on /api/?callback=foo and /api/whatever?callback=foo
RewriteRule ^api(?:/([^/]*))?$ index.php?api&url=$1 [QSA,L]
RewriteCond %{REQUEST_URI}?%{QUERY_STRING} !/api(/.*)?\?callback=.*
RewriteRule ^([^.]*)$ index.php?url=$1 [QSA,L]