How can we handle large matrices in matlab(larger than 10000x10000)

Abolfazl picture Abolfazl · May 25, 2011 · Viewed 9.8k times · Source

In my program I am faced with some matrices that are larger than 10000x10000. I cannot transpose or inverse them, how can this problem be overcome?

??? Error using ==> ctranspose
Out of memory. Type HELP MEMORY for your options.
Error in ==> programname1 at 70
    B = cell2mat(C(:,:,s))'; 
Out of memory. Type HELP MEMORY for your options.
Example 1: Run the MEMORY command on a 32-bit Windows system:


    >> memory
    Maximum possible array:             677 MB (7.101e+008 bytes) *
    Memory available for all arrays:   1602 MB (1.680e+009 bytes) **
    Memory used by MATLAB:              327 MB (3.425e+008 bytes)
    Physical Memory (RAM):             3327 MB (3.489e+009 bytes)

    *  Limited by contiguous virtual address space available.
    ** Limited by virtual address space available.

Example 2: Run the MEMORY command on a 64-bit Windows system:

    >> memory
    Maximum possible array:               4577 MB (4.800e+009 bytes) *
    Memory available for all arrays:      4577 MB (4.800e+009 bytes) *
    Memory used by MATLAB:                 330 MB (3.458e+008 bytes)
    Physical Memory (RAM):                3503 MB (3.674e+009 bytes)

==============================================================================

 memory
% Maximum possible array:            1603 MB (1.681e+009 bytes) *
% Memory available for all arrays:   2237 MB (2.346e+009 bytes) **
% Memory used by MATLAB:              469 MB (4.917e+008 bytes)
% Physical Memory (RAM):             3002 MB (3.148e+009 bytes)


I have used sparse for C. 

B = cell2mat(C);
clear C       %#  to reduce the allocated RAM
P=B\b;

Name         Size                  Bytes  Class     Attributes     

  B         5697x5697            584165092  double    sparse, complex
  C         1899x1899            858213576  cell                     
  b         5697x1                   91152  double    complex        

==============================================================================
??? Error using ==> mldivide
Out of memory. Type HELP MEMORY for your options.

Error in ==> programname at 82
    P=B\b; 

==============================================================================

Edit: 27.05.11

Name         Size                  Bytes  Class     Attributes

  C          997x997             131209188  cell   
  B            2991x2991             71568648  single    complex        
  Bdp          2991x2991            143137296  double    complex        
  Bsparse      2991x2991            156948988  double    sparse, complex

  Bdp=double(B);
  Bsparse=sparse(Bdp);

I used single precision, witch gave the same accuracy as in double precision

It's better, Am I right?

Answer

Edric picture Edric · May 25, 2011

A few suggestions:

  1. If possible, as @yoda suggested, use sparse matrices
  2. Do you really need the inverse? If you're solving a linear system (Ax=b), you should use MATLAB's backslash operator.
  3. If you really need huge dense matrices, you can harness the memory of several machines using distributed arrays and MATLAB distributed computing server.