Slow performance filtering file Post: 302495437

Sponsored Content

Top Forums Shell Programming and Scripting Slow performance filtering file Post 302495437 by Miila on Thursday 10th of February 2011 08:50:33 AM

02-10-2011

Registered User

Slow performance filtering file

Please, I need help tuning my script. It works but it's too slow.
The code reads an acivity log file with 50.000 - 100.000 lines and filters error messages from it. The data in the actlog file look similar to this:

Code:

02/08/2011 00:25:01,ANR2034E QUERY MOUNT: No match found using this criteria. (SESSION: 211433)
or
02/08/2011 01:05:50,"ANE4991I (Session: 211857, Node: xxx) blablabla (SESSION: 211857)"

I was trying to do the same with awk but it seems it's over my head. Any ideas will be appreciated. Thanks!

here is the code:

Code:

while read LINE
do
    A=$(echo ${LINE} | cut -c 21)
    if [[ ${A} == "\"" ]];then  
        MSGDATE=$(echo ${LINE} | cut -c 1-10)
        MSGTIME=$(echo ${LINE} | cut -c 12-19)
        PREFIX=$(echo ${LINE} | cut -c 22-24)
        MSGNUM=$(echo ${LINE} | cut -c 25-28)
        MSGTYPE=$(echo ${LINE} | cut -c 29)
        MESSAGE=$(echo ${LINE} | cut -c 30-)
        if [[ ${MSGTYPE} == "E" ]];then
            echo "${MSGDATE},${MSGTIME},${PREFIX}${MSGNUM}${MSGTYPE},${MESSAGE}" >> ${TMPFILE}
        fi
    else  
        MSGDATE=$(echo ${LINE} | cut -c 1-10)
        MSGTIME=$(echo ${LINE} | cut -c 12-19)
        PREFIX=$(echo ${LINE} | cut -c 21-23)
        MSGNUM=$(echo ${LINE} | cut -c 24-27)
        MSGTYPE=$(echo ${LINE} | cut -c 28)
        MESSAGE=$(echo ${LINE} | cut -c 29-)
        if [[ ${MSGTYPE} == "E" ]];then
            echo "${MSGDATE},${MSGTIME},${PREFIX}${MSGNUM}${MSGTYPE},${MESSAGE}" >> ${TMPFILE}
        fi
    fi
done <${ACTLOGFILE}

Miila

View Public Profile for Miila

Find all posts by Miila

8 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Help! Slow Performance

Is the performance now very, very slow (pages take a very long time to load)? Or is it just me? Neo

2. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other...

3. Filesystems, Disks and Memory

Slow Copy(CP) performance

Hi all We have got issues with copying a 2.6 GB file from one folder to another folder. Well, this is not the first issue we are having on the box currently, i will try to explain everything we have done from the past 2 days. We got a message 2 days back saying that our Production is 98%...

4. UNIX for Dummies Questions & Answers

Slow copy/performance... between volumes

hi guys We are seeing weird issues on my Linux Suse 10, it has lotus 8.5 and 1 filesystem for OS and another for Lotus Database. the issue is when the Lotus service starts wait on top is very high about 25% percent and in general CPU usage is very high we found that when this happens if we...

5. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%....

6. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system...

7. Solaris

Solaris 11.1 Slow Network Performance

I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are...

8. Filesystems, Disks and Memory

Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system -...

LEARN ABOUT SUNOS

echo

echo(3XCURSES)						  X/Open Curses Library Functions					    echo(3XCURSES)

NAME

       echo, noecho - enable/disable terminal echo

SYNOPSIS

       cc [ flag... ] file... -I /usr/xpg4/include -L /usr/xpg4/lib 
       -R /usr/xpg4/lib -lcurses [ library... ]

       c89 [ flag... ] file... -lcurses [ library... ]

       #include <curses.h>

       int echo(void);

       int noecho(void);

DESCRIPTION

       The  echo()  function enables Echo mode for the current screen. The noecho() function disables Echo mode for the current screen. Initially,
       curses software echo mode is enabled and hardware echo mode of the tty driver is disabled. The echo() and noecho() functions control  soft-
       ware echo only. Hardware echo must remain disabled for the duration of the application, else the behavior is undefined.

RETURN VALUES

       Upon successful completion, these functions return OK. Otherwise, they return ERR.

ERRORS

       No errors are defined.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Standard			   |
       +-----------------------------+-----------------------------+
       |MT-Level		     |Unsafe			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       getch(3XCURSES), getstr(3XCURSES), initscr(3XCURSES), libcurses(3XCURSES), scanw(3XCURSES), attributes(5), standards(5)

SunOS 5.10							    5 Jun 2002							    echo(3XCURSES)