Nix checks whether any player paid mana to cast the spell, not whether its current controller did. If Commandeer is cast on a spell, then Nix is cast on that spell, Nix won’t counter it unless the player who originally cast that spell spent no mana to cast it.

Nix doesn’t check whether a spell’s mana cost is 0. Rather, it checks how much mana was actually spent to cast it. It can counter a spell with mana cost 0 (such as Slaughter Pact), a spell cast via an alternative nonmana cost (such as Allosaurus Rider) or alternative cost of {0} (such as Basking Rootwalla), a spell whose cost to cast it was reduced to 0 (such as Frogmite), a spell cast without paying its cost due to an effect (such as suspend or Mindleech Mass’s ability), or a copy of a spell, among other possibilities. It will not counter a spell that had an additional or alternative mana cost paid to cast it, regardless of that spell’s mana value (such as a face-down spell cast with morph, or an Ornithopter cast while Trinisphere is on the battlefield, for example).

Nix can target any spell. The condition is checked only on resolution.