Is it possible to enable the UserDir Directive for just one VirtualHost rather than have it on for all and then disable it (with "UserDir disable") for each VirtualHost you don't want it on?

I have tried by putting this inside a <VirtualHost> and comment out everything in the global config (/etc/apache2/conf.d/userdir.conf). No luck though.

<IfModule mod_userdir.c>
    UserDir public.www
    UserDir disabled root

    <Directory /home/*/public.www>
            AllowOverride FileInfo AuthConfig Limit Indexes
            Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
            <Limit GET POST OPTIONS>
                    Order allow,deny
                    Allow from all
            </Limit>
            <LimitExcept GET POST OPTIONS>
                    Order deny,allow
                    Deny from all
            </LimitExcept>
    </Directory>
</IfModule>