Create a CUstom Archive by Year, but just for a Single Category

I have a request from my designer to create a list of links to a page that only shows posts published by year and that belong to a specific category.

So here is the design:

I’ve found a function that can force my Archive to be just by year

/**
 * Load Archive Widget by Year
 */

function my_limit_archives( $args ) {
    $args['type'] = 'yearly';
    return $args;
}

add_filter( 'widget_archives_args', 'my_limit_archives' );
add_filter( 'widget_archives_dropdown_args', 'my_limit_archives' );

This is fine, as it does show what I want, although those links all push to just www.mysite.com/2018 to show ALL posts published in 2018.

The problem is I need a version of this which only shows a specific category, not ALL posts. So like www.mysite.com/{category-name}/2018