Fast(er) way to get folder size with batch script

windows batch-file cmd du
StanM picture StanM · May 28, 2015 · Viewed 13.9k times · Source

PLEASE SEE BELOW THE ORIGINAL QUESTION FOR SOME TEST COMPARISONS OF DIFFERENT WAYS:

So I tried 2 ways so far:

1.Iterate through directory using the code from Get Folder Size from Windows Command Line : 

@echo off
set size=0
for /r %%x in (folder\*) do set /a size+=%%~zx
echo %size% Bytes

2.Save output of a 

'dir %folder% /s /a'

into a text file, and then read in the size at the bottom

3.The last way I am trying right now is using du (disk utility tool from MS - https://technet.microsoft.com/en-us/sysinternals/bb896651.aspx ).

Now with exception of #3 both of those ways seem way too slow for what I need (100s of thousands of files). So the question is which one of these is the fastest / should be fastest, and if there are any other fast(er) ways to get size of folder contents that has 100k+ files (and there are 100s of folders)

START EDIT:

Below is my very hacky way of doing the comparison (butchered my program to see some outputs)
There are some small bugs with some parts like option 3 will fail because it tries to handle a number bigger than 32-bit limit, and I'm sure there is some more issues, but the general timings I think are evident unless I really messed up on my logic.

Option I: Iterate through directories, using VB script to read in the text output from 'dir' and look for the size at the end + convert it to MB (originally got it from somewhere else that I actually lose the place where I got it from) Option II: Iterate, with findstr pipe and output the result directly (no converstion to MB) - from @MC ND Option III: use the compact command to iterate - from @npocmaka Option IV: from @user1016274 - using robocoby

(There are some more answers, but these are the ones I've been able to incorporate)

These are the results i got, and they are pretty consistent in relevance to each other, robocopy blows them away

Option I and Option II were usually close, with Option II slightly better (anywhere from 1min 10sec to 2min 10 secs for both, not sure where the difference is coming from) Part III - 16-17 mins Part IV - 10-20 seconds

@echo OFF
setlocal enabledelayedexpansion

REM OPTION I - directory iteration
REM OPTION II - iteration with findstr pipe
REM OPTION III - compact

:MAIN
REM Initialize log filename
for /f "delims=" %%a in ('echo %date:~10,4%%date:~4,2%%date:~7,2%%time:~0,2%%time:~3,2%%time:~6,2%') do @set LOGFILEPOSTFIX=%%a
set LOGFILEPOSTFIX=%date:~10,4%%date:~4,2%%date:~7,2%%time:~0,2%%time:~3,2%%time:~6,2%
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% 
set "LOGFILE=Proj_not_in_db_%LOGFILEPOSTFIX%.log"


set option=1
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART I ---- Directory Listing into file, iterate through the sizes of all files inside folder >> %LOGFILE%
echo %TIMESTAMP% - PART I
call :PROCESSFOLDER
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART I ---- END >> %LOGFILE%
echo %TIMESTAMP% - PART I - END
set option=2
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART II  findstr pipe ---- >> %LOGFILE%
echo %TIMESTAMP% - PART II
call :PROCESSFOLDER
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART II ---- END>> %LOGFILE%
echo %TIMESTAMP% - PART II - END
set option=3
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART III compact ---- >> %LOGFILE%
echo %TIMESTAMP% - PART III
call :PROCESSFOLDER
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART III ---- END>> %LOGFILE%
echo %TIMESTAMP% - PART III - END
set option=4
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART IV robocopy ---- >> %LOGFILE%
echo %TIMESTAMP% - PART IV
call :PROCESSFOLDER

call :CLEANUP
echo FINAL
pause
goto :EOF

:PROCESSFOLDER

    echo C:\Windows
    echo Processing C:\Windows >>  %LOGFILE%
    break > projects_in_folder.tmp
    for /f "tokens=1-4,* SKIP=7" %%b IN ('dir "C:\Windows" /Q /TW /AD') do (
        set _folder=%%f
        REM Don't write the 2 lines at the end displaying summary information
        if NOT "%%e" EQU "bytes" (
            SET _folder=!_folder:~23!
            echo !_folder!,%%b>> projects_in_folder.tmp
        )   
    )
    set "folder_path=C:\Windows"
    call :COMPARE
goto :EOF

:COMPARE
set file_name=%folder_path:\=_%
break > "%file_name%.txt"
if %option%==4 (
    set "full_path=C:\Windows"
    call :GETFOLDERINFO4
    set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
    echo %TIMESTAMP% - PART IV ---- END>> %LOGFILE%
    echo %TIMESTAMP% - PART IV - END
)


for /f "tokens=1,2* delims=," %%a in (projects_in_folder.tmp) do (
    for /f "tokens=1,* delims=_" %%x in ("%%a") do (
        set "projcode=%%x"
    )
    set full_path=%folder_path%\%%a
    if %option%==1 call :GETFOLDERINFO 
    if %option%==2 call :GETFOLDERINFO2
    if %option%==3 call :GETFOLDERINFO3

    echo PROJ: %%a SIZE: !totalsize! LASTMODIFIED: %%b >> %LOGFILE%
)
goto :EOF

:GETFOLDERINFO2
set "size=0"
set target=!full_path!
for /f "tokens=3,5" %%a in ('
    dir /a /s /w /-c "%target%"
    ^| findstr /b /l /c:"  "
    ') do if "%%b"=="" set "size=%%a"
echo %size%
set totalsize=%size%
goto :EOF

:GETFOLDERINFO4
pushd "%full_path%" || goto :EOF
setlocal

for /f "tokens=1-10,* delims= " %%a in ('
    robocopy %full_path% %TEMP% /S /L /BYTES /XJ /NFL /NDL /NJH ^| find "Bytes"
') do echo %full_path%: %%c
popd    
goto :EOF

:GETFOLDERINFO
set totalsize=0
dir "%full_path%" /s /a > size.txt 
REM Run VBScript that outputs size in MB which is saved
pushd %~dp0
start /b "" cscript /nologo foldersize.vbs
FOR /F "usebackq tokens=*" %%r in (`CSCRIPT "foldersize.vbs"`) DO SET totalsize=%%r
echo bla > nul
goto :EOF

:GETFOLDERINFO3
set "last=#"
set "_size="
for /f "tokens=1 delims= " %%s in ('compact /s:"%full_path%" /q ') do (
        set "_size=!last!"
        set "last=%%s"
)
set "_size=%_size:  =%"
set "_size=%_size: =%"
set "_size=%_size:.=%"
set "_size=%_size:,=%"
set "_size=%_size:      =%"
echo folder size is : %_size% bytes
set totalsize=%_size%
goto :EOF


:CLEANUP

DEL /Q /S projects_in_folder.tmp
DEL /Q /S size.txt
goto :EOF

Answer

MC ND picture MC ND · May 28, 2015

You can try with (in the spirit of your second case)

@echo off
    setlocal enableextensions disabledelayedexpansion

    set "target=%~1"
    if not defined target set "target=%cd%"

    set "size=0"
    for /f "tokens=3,5" %%a in ('
        dir /a /s /w /-c "%target%"
        ^| findstr /b /l /c:"  "
    ') do if "%%b"=="" set "size=%%a"

    echo %size%

Related questions

How do I get current date/time on the Windows command line in a suitable format for usage in a file/folder name?

Update: Now that it's 2016 I'd use PowerShell for this unless there's a really compelling backwards-compatible reason for it, particularly because of the regional settings issue with using date. See @npocmaka's https://stackoverflow.com/a/19799236/8479 What's a Windows command line statement(…

Read More
Batch file to delete files older than N days

I am looking for a way to delete all files older than 7 days in a batch file. I've searched around the web, and found some examples with hundreds of lines of code, and others that required installing extra command line …

Read More
How to delete files/subfolders in a specific directory at the command prompt in Windows

Say, there is a variable called %pathtofolder%, as it makes it clear it is a full path of a folder. I want to delete every single file and subfolder in this directory, but not the directory itself. But, there might …

Read More