Colleges in Australia can just be private schools being a littler fancier. But then there's Colleges like Western Sydney College which is a break off the University of Western Sydney...

I use uni because I go to a uni, not a college

To me university is what they call college in the UK

OMG, I came in here thinking I'd get some answers, and I'm just more confused than ever.

Anyway, here's some "U.N.I." for you to enjoy.

University in the UK is what the US would call college
College in the UK is a separate thing (usually after high school but before university) with academic and vocational courses.

They are used interchangeably where I live.

Though I suppose "college" is applied to trade schools while Uni is not.