Creating lots of data quickly with Laravel

I wrote this article on Wednesday, September 9, 2020

Creating data using factories is a great convience when working with seeding test databases - but its very slow for big datasets you might want to create. Lets speed it up.

TLDR; Code for the artisan command is at the bottom of the post

Lets take for example a simple factory create command and run it a 1000 times. We'll do it in an artisan command we created for the purpose of creating all this data so we can track times, but the main creating code is:

Model::factory()->count(1000)->create();

$ php artisan create:data 1000
Adding 1,000 Rows of data
✔ Done (2.99825s)

Done in 3 seconds. We can do better.

What we are going to do is batch our inserts into bundles and instead of making 1,000 inserts like the create command does, we'll just make one every 500 (feel free to play and change this number!). The database engine doesn't mind big queries like this, and as you'll see, its much quicker (a factor of 10 usually!)

public function createChunked($rows)
{
    $chunk = 500;
    $chunks = ceil($rows / $chunk);

    //-- Loop through each chunk
    foreach (range(1, $chunks) as $cNum) {

        //-- Empty array to add stuff to
        $insert = [];

        //-- Create specific amount of rows if last chunk
        if ($cNum == $chunks) {
            $chunk = $rows - ($chunk * ($cNum - 1));
        }

        //-- Loop
        foreach (range(0, $chunk) as $n) {
            $p = Model::factory()->make();
            $d = $p->toArray();

            //-- Watch out - dates might not insert 
            //-- wire:correctly, so parse them before adding
            $d['created_at'] = Carbon::parse($d['created_at'])
            ->format("Y-m-d H:i:s");
            $insert[] = $d;
        }

        //-- Insert our array
        \DB::table('model_table')->insert($insert);
    }
}
$ php artisan create:data 1000
Adding 1,000 Rows of data
✔ Done (0.28939s)

Done in 1/10 of the time at 0.3 seconds 🔥🔥🔥

Code for the artisan command I created is below - just change the model and table used and its good to go!

Sorry, the Github Gist embed is a bit wonky on mobile - check out the page here: https://gist.github.com/JakePrice86/4a30e6c9e738a6b164822a58f81e6a09

Created with Jigsaw, hosted on Netlify.