Today I need to import from a CSV file about 14500 records. Doing it the old way isn't a great idea, the server will be overloaded and the result may be unesxpected!

Here Loops take the field, let's figure out how with an example.

In Lipsiadmin create Loops is very simple, in the root of your project:

$ script/generate loop
a new directory "loop" is created under "app" and a new configuration file "loops.yml" in the "config" directory.

You will also find a "simple_loop.rb" file that contains a loop with a debug line to test the loop.

The config file contains some variable, the most important is poll_period, change it with the second you want your loop to start.

global:
    logger: stdout
    poll_period: 30
    workers_engine: fork

Now in the console you can start your loop in foreground with: 

$ script/loops -a
for a list of available commands: 
$ script/loops -h
or reference to Lipsiadmin API.

So we are able to make a loop, let's see a good example on how to use it.

In my case i need to import a list of accounts, so create an account_import model

$ script/generate model account_import

we add to the migration: timestamps log and completed_at

class CreateAccountImports < ActiveRecord::Migration
    def self.up
      create_table :account_imports do |t|
        t.text :log
        t.datetime :completed_at
        t.timestamps
      end
    end

    def self.down
      drop_table :account_imports
    end
  end

then we generate the lipsiadmin backend page

$ script/generate backend_page account_import

and add to the model an attachment, remember to generate the Lipsiadmin attachment if you did't do it yet.

class AccountImport < ActiveRecord::Base
  has_one_attachment :client, :dependent => :destroy
end

and modify the "app/viwe/backend/account_imports/_form" with the file field:

...
-tab :general do
  %table
    -unless @account_import.new_record?
      File Name:
      =link_to @account_import.client.attached_file_name, @account_import.client.url
      ="( #{number_to_human_size(@account_import.client.attached_file_size)} )"    
      
    .box=file_field_tag "account_import[client_attributes][file]"

remember that for file upload you need to add multipart => true to new and edit pages

-form_tag({:action => :create}, :method => :post, :multipart => true) do
  =render :partial => "form"

The upload part is complete, now we need to add the part that take the CSV file, parse it and create the accounts.
Guess who do this? Loops

Create a new file named "acount_import_loop.rb" in "app/loops"

class AccountImportLoop < Lipsiadmin::Loops::Base
  def run
    # we search for the first record that isn't completed
    if import = AccountImport.first( :conditions => "completed_at IS NULL")
      import.update_attributes( :log => "Import Running...") # update the log 
      file = "#{Rails.root}/public" + import.client.url  # set file variable with uploaded file position
      csv = FasterCSV.read(file, { :col_sep => ";", :skip_blanks => true }) # I used FasterCSV for parsing
    
      import_log = "Import of #{csv.size} Client Started"
      import_log << "---------------------------------------"
      
      # the cycle that import accounts
      csv.each_with_index do |row, index|
          
        account = Account.find_by_client_code(row[0].strip) || Account.new
  
        account.client_code = row[0].strip
        ....
        # Row matching
        
        if !account.save
          import_log << "Import Client #{index+1}/#{csv.size} FAILED"
        end
      end
        
      import_log << "---------------------------------------"
      import_log << "Import of #{csv.size} Clients Done"
      import.update_attributes( :log => import_log, :completed_at => Time.now )
    end
  end
end

Add this loop to configuration file "/config/loop.yml"

....
  loops:
    account_import_loop:
      workers_number: 1

The part that do the import is complete, we can test it by creating a new AccountImport record.

To test the loop start it manually with:

$ script/loops -a

In production we can't start the loop manually, so we can create a task to do the job. Create a file in "/lib/taks/import.rake"

namespace :loop do

    desc "Start import loop"
    task :start => :stop do
      #start all loops in background as daemons
      exec "#{RAILS_ROOT}/script/loops -a -d"
    end

    desc "Stop import loop"
    task :stop do
      system "#{RAILS_ROOT}/script/loops -s"
      puts "Stop Loop... DONE"
    end

end

Tip: When you try to stop the loop the program wait one cycle, the one defined in the poll period, so if you set an high poll, for example 3600 you need to wait that period before the loop stop.

That's all! We have a loop started by a rake task, that search if there is a database record with an uploaded CSV of clients every 30 seconds and create or update Accounts definitions.