Well, flip the tables around and imagine it’s a fanfic with YOU as the central character. It’d be a fictional narrative putting events/actions into your own personal history/future, yeah, that’d be disturbing. (And if it’s derogatory/slanderous, the author opens themselves up to a lawsuit too.)
Normally I bend over backwards to NOT associate my stories with real people/buildings/etc. So, I’ll make up town names, or at least scramble the road/street name the homes are on to avoid the Goonies House effect. It’s a fine line, but as a viewer, I know I was irritated when a show like Dead Like Me films in Vancouver BC and tries to pass it off as Seattle… it broke the immersion because I knew it was off.