It is very unlike the source is Connections. Only two pages would be impact by Connections in any meaningful way assuming your only have a single frontend directory page. The first page would be that frontend directory page. The other page would be the Connections : Manage page which is no more resource intensive than the WordPress Posts or Pages admin page.
There is one scenario in which you could exhaust serve resources. On the frontend directory page, if you are displaying a lot of entries. I recommend 20 for optimal performance though many users will load 50 to a 100 per page without performance being affected too much. This is not a Connections limitation of bug as the same would occur if one would try to load many WordPress posts on a single page.
You may have to better optimize your WordPress installation such as deactivating unnecessary plugins increase the PHP memory limit to 128MB or 256MB and change the max PHP execution to 300 seconds. If the PHP version that is being run is old such as 5.2–5.6 I very highly recommend changing it to 7.1. WordPress actually recommends 7.2 but most plugins have not yet caught up to 7.2 since it has only recently been released. Most web hosts provide a simple method in their control panel to switch PHP versions. If you are not running PHP 7.1 and switch to it you will receive a large performance boost while decreasing the required server resources.
Lastly, look at a caching plugin. I recommend WP Rocket. It’s simple, well support and generally just works.
Hope this helps, let me know what you learn.