Well, I need to create vertex buffers during loading for each material and octree node so one extra indirection during loading isn't going to be a big problem. And in fact it's probably going to be faster to sort the indicies in the octree by material than it is to sort the actual polygons objects.
I realise that in opengl you don't actually need to create vertex buffers, you can just call glvertex for each point, but you're going to want to at some point
EDIT: But it probably doesn't matter, it's easy enough to load the polygons into an array if I want them that way.