In NetSim v10.0 (or above), 


OSPF cost calculations are done in SendLSU.c file which is part of the Routing Project. 


In function fn_NetSim_OSPF_LSUpacketformation() the lines:
dBandwidth=NETWORK->ppstruNetSimLinks[nLink_Id-1]->puniMedProp.pstruWiredLink.dDataRateUp;nCost=(int)round(REFERENCE_BANDWIDTH/dBandwidth);


are related to the cost calculations. 



The cost variable ncost is further used in the same functions in the lines:
pstruNewEntry = calloc(1,sizeof(struct stru_RouterLSA_Entry));pstruNewEntry->nMetric = pstruRoutingTable->nCost+nCost;


You can modify these lines suitably to calculate Metric in a different way.


Regarding the calculation of cost, you can refer to Experiment 12 of NetSim Experiment Manual.