Boil brown sugar, sugar, milk and butter together for 5 minutes. Stirring constantly as it scorches easily. Pour mixture over crackers. Top with another layer of graham crackers.
For frosting, melt together peanut butter and chocolate chips. Keep bars in refrigerator.