The two-time Grammy winner revealed that he and Swift were once "working on something," but it didn't come to fruition ...