I haven't entirely followed the advice to stay away. I actually like dropping in once a week or so to change up my work space. I can only write at my kitchen table for so long. It's nice to stop in, grab mail, chat with colleagues, say hi to students, do some work, and scamper back home. Last week I took my four-year-old for a campus stroll, something I normally don't have time to do. And it's nice talking with people without being in a hurry. Stop and smell the academy, I guess!
As my sabbatical continues, I plan to go to campus sporadically to grab lunch with friends, talk with students and colleagues, and do work in my office or at the library. In my experience so far, I haven't felt like staying away entirely. But my colleague was exactly right: I'm not that important. So I will continue to do what's worked for me: mostly stay at home, write and read at my kitchen table, and enjoy a mix of productivity and relaxation.