Oligarth despawns and spawns in multiple places, so you want to clear them out before beginning. When you first attack him, he will despawn around 50-60% and respawn inside the small room to the southeast of him.
When you engage him, if your dps is not high enough then he will dispel (including removing discs) everyone on his hate list. After bringing him down to 50%, he will despawn and respawn in the room adjacent to there. Same deal here, except you have to take him down to about 25-30%.
After that the third despawn, he respawns in the altar room that is listed as "Spiritseeker Nadox" on the map. Same thing as before.
