Thanks for the report, your description pointed straight at it. Fixed in 3.4.2 (just released).
The companion kept its last job in memory after finishing it. With the long Construction Range (default 1000 tiles), it never forgot that spot: as soon as it was about 100 tiles away from it, it flew back there, then returned to you, and so on. That is the "point where I built something in the past" you noticed. A finished job is now forgotten right away.
While I was in there, I also fixed the deconstruction planner and blueprints placed far away (from the map view, for example): the marked area was never actually queued, so the companion only found it if it was close to you. It now goes there directly.
If you still see it after updating, tell me roughly how far you were from the old build site and I'll dig further.