Drop using pandas for ParticleTable

Part of #397

Pandas is a very heavy import (taking ~1.6 seconds) but it's only used for a relatively bit of work when loading the particle table.

This PR drops the use of pandas to avoid having this fixed overhead.