Last friday I collided to quite frustrating moment while doing user import to clients Midgard site.

I created a simple import script to parse the old data from CSV and insert them as Midgard persons. After the script runned for a while I started seeing errors that the MySQL wasn't responding. Checked the server and realized that all disks were mounted as read-only and I couldn't mount them back. So after weekend of strugling with the server we decided with the hosting company to move all the data to new server.

After this I made some minor modifications to the import script so it sleeps 2 minutes after every 250 imported users. And monitored the server all the time. After 5hours of importing everything was ready and the server is still doing ok.

So just a small advice to anyone who has to do big imports (over 18k objects), add some sleep time to the import script so the server can take a deep breath once in a while.

on the next Midgard dev meeting I'll try to do this again on a development server and trace for memory leaks.

Midgard