When a SharePoint Access Control List (ACL) size exceeds more than 64kb for a particular URL, the contents under that URL become unsearchable due to limitations in the SharePoint search engine.  The error most often seen is The Parameter is Incorrect which really helps to pinpoint the problem (its difficult to convey extreme sarcasm here, please note that it is intended).  Exceeding this limit is not unheard of – it can happen when users brute force security into working by continually overriding inherited permissions and assigning user-level access to securable objects.

Once you have this issue, determining where you need to focus to fix the problem can be difficult.  Fortunately, there is a query that you can run on a content database that can help identify the issue:

This query results in a list of ACL sizes and entry counts on a site-by-site basis.  You can also remove grouping to see a more granular breakdown:

SELECT [ScopeUrl] AS URL, 
     SUM(DATALENGTH([Acl]))/1024 as AclSizeKB,
     COUNT(*) AS AclEntries
FROM [Perms] (NOLOCK)
GROUP BY ScopeUrl
ORDER BY AclSizeKB DESC