If no creature is blocking or being blocked by Vortex Elemental as its first ability resolves, it alone will be put on top of its owner’s library, and that library will be shuffled. You can activate the first ability outside of combat.

If Vortex Elemental isn’t on the battlefield when its first ability resolves, any creatures blocking or blocked by it when it left the battlefield will be put on top of their owners’ libraries and those libraries will be shuffled.

The creature blocks Vortex Elemental only if it’s able to do so as the declare blockers step begins. If, at that time, the creature is tapped, it’s affected by a spell or ability that says it can’t block, or Vortex Elemental isn’t attacking its controller or a planeswalker controlled by that player, then it doesn’t block. If there’s a cost associated with having the creature block, the player isn’t forced to pay that cost. If that cost isn’t paid, the creature won’t block.