Remove two features added upstream
Importing the full library would double the jar size... its way too large. So lets just import the basic int/long based collections to then use to improve performance on these kind of collections.
Objects loaded over different class loaders are not the same. Nasty dragons lie here. Also clean up the previous patch to no longer butcher imports.