Visiting Kerlingarfoss should be combined with visiting Svöðufoss as they're right next to each other. Two very cool waterfalls nearby each other, often overlooked and not super easy to find, so you won't find lots of people here. Our navigation wasn't too helpful, but about 3-4km east of Ólafsvík, take a left turn from the main road on to a gravel road (GPS coords: 64.9053, -23.7855). Follow this road and bend left after about 1.3km, after another 300m you will find the parking for both waterfalls.
To get to Kerlingarfoss follow the foot path leading South-West until you get to a stone wall. From there head South-East, jump across the smaller stream and head South while keeping the bigger stream on your right. This will bring you straight to the waterfall. You can follow the path up next to the waterfall to get a real close look.