I have put the Openfire.jar in the local Maven repository and setup Maven to build my plugins (with Maven assemblies/descriptors).
In these plugins I've referenced Openfire.jar with scope "provided". The CI is happy with it.
I debug plugins together with the Openfire source code, i.e. start Openfire from IDE, build the plugin first with Maven and copy it in the $openfireHome/plugins folder. My plugin is an IntelliJ module then, so IntelliJ catches it up and let me debug.
I made a few changes to my Openfire source code (rewrote the PluginManager logic) and now i dont need to copy the jars to the pluginsDir
But the idea we are looking for is almost what you are doing