With a gem called activerecord-import, importing a huge amount of records into mysql table becomes an easier task to do, and this post demonstrates how to achieve it.
Suppose we have a mysql table called visitor_ids that saves the visitor ids for some really high traffic website. The SQL to create such table:
And an active record model is defined as VisitorId.
Now we have say more than 1 million such visitor ids needed to be inserted into this table.
Use CSV as the data source
Use activerecord-import for bulk import
initialize a queue to buffer the record objects
produces the records
imports the records
run the import
How long does it take to import 1 million records?
As a reference though:
It took around 2 minutes for an AWS instance with 16G memory and 4 CPUs to import about 1 million records into a mysql instance also hosted by AWS.