To change the size of the font 2 steps need to be done:

1. Add as Style for the Tablayout
2. Apply the style to the TabLayout in the XML

Find or create res -> values -> styles.xml

1. Add the Style to the XML

<style name="MyCustomTabText" parent="TextAppearance.Design.Tab">
    <item name="android:textSize">12sp</item>
</style>

2. Apply Style

Find the Layout containing the TabLayout and add the style. The added line is bold.

<android.support.design.widget.TabLayout
    android:id="@+id/tabs"
    app:tabTextAppearance="@style/MyCustomTabText" 
    android:layout_width="match_parent"
    android:layout_height="wrap_content" />