PyMOL's current “sweet spot” goes up to about 50,000 atoms assuming that the software has two gigabytes (GB) of heap space to work with on a 32-bit operation system. On a 64-bit operating system with 8-16 GB of RAM, PyMOL should be able to handle up to several hundred-thousand atoms at once.
Visualization of million-atom systems is possible, but you have must be very selective in terms of the features you employ. We recommend using Linux, and suggest launching the software with the -O 1 option in order to force each atom to be represented by a single pixel.
./pymol -O 1 million_atom_system.pdb
If you have a graphics card with shader support, you can use -O 5 instead in order to get pixel-perfect atomic spheres.
./pymol -O 5 million_atom_system.pdb
As for ray-tracing million atom systems, it takes about 2 GB of RAM total to render a million atoms using spheres, so we recommend using 64-bit Linux builds for anything beyond that.
If working with molecular dynamics trajectories, PyMOL's best case memory usage is 16 bytes per 3D atomic coordinate, which works out to 1,000 frames of a 50,000 atom molecular dynamics trajectory per GB of available RAM (or 10,000 frames of a 5,000 atom trajectory per GB). Actual RAM usage will be worse than that, depending upon what representations are shown.
Note: for optimal performance when playing back trajectories with PyMOL versions 1.2 and beyond
set defer_builds_mode, 5
before loading your trajectory. This reduces RAM consumption dramatically and can also boost playback performance to VMD-like levels.