Yes, loading hundreds of entries on a single page will indeed load more slowing. This is not a bug or limitation in Connections. If you were to attempt to load hundreds of WordPress posts on a single page it probably would not load at all which is way there limit posts to 5 per page by default. What you need with your large directory is a premium template, all of which support paginating the directory, 20 per page. All premium templates support pagination but some have this disabled by default since their focus on small eam directories. It you like one of those, it is very simple to enable pagination.
Hope that answers your question!