Display all posts from all categories with pagination

Alright, this might be a tricky one to explain but I’ll do my best.

At the front page I want to display all the posts from all the categories. However I want them to be divided. Take this for an example:

The years are actually categories. Each category represents a different year. So I insert my posts (which only display the thumbnail image in the loop) in the category that applies to when the photos were taken.

At the start of each category I want it to show the category’s name, as displayed above. I’m also using the jQuery infinite-scroll from Paul Irish, so basically the pagination buttons are replaced with the loading bar, and it just loads at the same page when the user scrolls down.

My current code is:

<div class="post clear">
    <a href="<?php echo get_category_link(4); ?>"><div class="date"><?php echo get_cat_name(4); ?></div></a>
    <?php
    $catPost = get_posts('cat=4&posts_per_page=-1');
    foreach ($catPost as $post) : setup_postdata($post); ?>
        <?php get_template_part('content'); ?>
    <?php endforeach;?>
</div>
<div class="post clear">
    <a href="<?php echo get_category_link(1); ?>"><div class="date"><?php echo get_cat_name(1); ?></div></a>
    <?php
    $catPost = get_posts('cat=1&posts_per_page=-1');
    foreach ($catPost as $post) : setup_postdata($post); ?>
        <?php get_template_part('content'); ?>
    <?php endforeach;?>
</div>

And it’s actually working. Well, almost. First of all I have to insert category by category everytime I create a new one, with most of the code being duplicated. The only thing that changes are the categories’ IDs. It would be much better if with only one code I could retrieve all categories the way I want.

Also, another problem, is that pagination doesn’t work. If I scroll down (or disable infinite-scroll and click on the second page) it’ll just show repeated content over and over again.

I think you guys got the idea. I want users to be able to browse my gallery and everytime a new set of photos from a different year pops up, the date will appear on the first image. I came up with this solution of categories but it’s becoming a little hard to make it work the way I want.

Any solutions/suggestions?

Thanks in advance!

EDIT @bestprogrammerintheworld

<?php
    $args=array(
      'posts_per_page' => -1,
      'orderby' => array( 'date' => 'DESC', 'title' => 'ASC' ),  //This works in WP4.0!!!     
      'caller_get_posts'=> 1
    );
    $my_query = new WP_Query($args);


    if( $my_query->have_posts() ) {
      $nr_datesyear = 0;
      while ($my_query->have_posts()) : $my_query->the_post();

        $year = get_the_date('Y');


        if ($year !== $yeargroup && $nr_datesyear <=5) {
            $nr_datesyear = 0;  
            echo '<div class="post clear">';
        }
        echo '<a href="'. $year .'"><span class="date">'. $year .'</span></a>';                  
        get_template_part('content');

        if ($year !== $yeargroup && $nr_datesyear <=5) {
            echo '<div>'; //end post year group
            $yeargroup = $year;   
        }

        $nr_datesyear++;  
            endwhile;            
        }

        wp_reset_query();
?>