Yes. Maybe.
Swinging it into view occasionally is unlikely to do harm, and some lenses are less likely to do harm than others, but a lens could focus the sun's image on some inner part of the camera and damage it. Think back to kids and magnifying glasses torturing insects and burning wood.
If It is just swinging on a strap, it is quite unlikely, as you would not get much dwell time. It's also unlikely the lens would actually be focused at any one instant in time for where the spot landed (e.g. focused at infinity focuses on the sensor, but the shutter is in front of that and would be out of focus). But you could.
And a lens cap is a good idea when not taking images for all sorts of reasons, this is just one more.
I have a slash burned through a SB800 flash where I had a Better Beamer (essentially a lens for a flash) on a tripod, with the sun out of camera lens view, but still in view of the fresnel on the flash. Focused the moving sun and burned a gouge in the plastic.
Also, be aware that stills with the sun in view (people often do it with very wide angles) get negligible dwell time on the sensor, but people shooting video have the shutter wide open and may get minutes of heating from dwell time. Again, depending on lens may do no harm at all, but look back at some horror stories from last summer's eclipse. Including burned shutter screens, so it's not just the sensor to worry about.
I wouldn't obsess over it (I don't obsess over it), but some basic precautions are a good idea -- lens cap when not in use, sun shade (helps with off-axis hot spots), and basically if it hurts your eyes when you look through the viewfinder it's too bright.
Ask anyone who has done any astro-photography; they greatly respect what the sun's focused light can do to equipment.