09-09-2012
deleting dupes in a row
Hello,
I have a large database in which name homonyms are arranged in a row. Since the database is large and generated by hand, very often dupes creep in. I want to remove the dupes either using an awk or perl script.
An input is given below
Quote:
mohammed=mohd=muhammed=mohammed=mohd=muhammed=md=muhmd
mahendra=mahendera=mahndra=mahendra=mahendera
The expected output is given below:
Quote:
md=mohammed=mohd=muhammed=muhmd
mahendera=mahendra=mahndra
As can be seen all the dupes are cleaned out.
At present I am using a macro which converts row to line, sorts and deleted dupes and restores the row structure. Since the database is huge, the macro takes a very long time.
Many thanks in advance for a speedy solution
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I need to delete all occurences of the repeated lines from a file and retain only the lines that is not repeated elsewhere in the file. As seen below the first two lines are same except that for the string "From BaseLine" and "From SMS".I shouldn't consider the string "From SMS" and "From... (7 Replies)
Discussion started by: ragavhere
7 Replies
2. Shell Programming and Scripting
Hi
Is it possible to do the following in a single command
/usr/xpg4/bin/sed -e '/rows selected/d' /aemu/CALLAUTO/callauto.txt > /aemu/CALLAUTO/callautonew.txt
/usr/xpg4/bin/sed -e '/^$/d' /aemu/CALLAUTO/callautonew.txt > /aemu/CALLAUTO/callauto_new.txt
exit (1 Reply)
Discussion started by: aemunathan
1 Replies
3. UNIX for Dummies Questions & Answers
How can you delete a row if a certain column is bigger than a certain number?
I have the following input:
20080709 20081222 95750 1 0 0.02 94.88
20080709 20081222 95750 2 0 0.89 94.88
20080709 20081222 9575 1 0 0 94.88
20080709 20081222 9575 2 0 0 94.88
20080709 20081222 9587.5 1 0 0... (1 Reply)
Discussion started by: Pep Puigvert
1 Replies
4. UNIX for Dummies Questions & Answers
How can you delete a row if a certain column is bigger than a certain number?
I have the following input:
20080709 20081222 95750 1 0 0.02 94.88
20080709 20081222 95750 2 0 0.89 94.88
20080709 20081222 9575 1 0 0 94.88
20080709 20081222 9575 2 0 0 94.88
20080709 20081222 9587.5 1 0 0... (6 Replies)
Discussion started by: Pep Puigvert
6 Replies
5. Shell Programming and Scripting
Hello,
I want to add a new row to a file to insert data without deleting the previous data there.
Example:
file
a
b
c
d
Output
a
b
newtext
c (6 Replies)
Discussion started by: joseamck
6 Replies
6. Shell Programming and Scripting
I am compiling a synonym dictionary which has the following structure
Headword=Synonym1,Synonym2 and so on, with each synonym separated by a comma.
As is usual in such cases manual preparation of synonyms results in repeating the synonym which results in dupes as in the example below:... (3 Replies)
Discussion started by: gimley
3 Replies
7. Shell Programming and Scripting
Hi,
I have a file which consists of two columns but the first one can be varying in length like
123456789 0abcd
123456789 0abcd
4015 0 0abcd
5000 0abcd
I want to go through the file reading each line, count the number of characters in the first column and delete... (2 Replies)
Discussion started by: swasid
2 Replies
8. Shell Programming and Scripting
Hello,
I have a very large file for which I would like to remove all rows for which the value of columns 2-5 is zero.
For instance I would like this file:
contig1, 0, 0, 0, 0
contig2, 1, 3, 5, 0
contig3, 0, 0, 0, 0
contig4, 0, 5, 6, 7
To become this file:
contig2, 1, 3, 5,0 ... (17 Replies)
Discussion started by: mouchkam
17 Replies
9. Shell Programming and Scripting
Hi, I want to move a new row to another table if the field from new row doesn't have the first word that I categorized (like: IRC blablabla, PTM blablabla, ADM blablabla, BS blablabla).
I already use this script but doesn't work as I expected.
CHECK_KEYWORD="$( mysql -uroot -p123456 smsd -N... (7 Replies)
Discussion started by: jazzyzha
7 Replies
10. Shell Programming and Scripting
Hi, I already succeed moving a new row to another table if the field from new row doesn't have the first word that I categorized (like: IRC blablabla, PTM blablabla, ADM blablabla, BS blablabla).
But it can't delete the old row. Please help me with the script.
my php script:
INSERT INTO... (2 Replies)
Discussion started by: jazzyzha
2 Replies
LEARN ABOUT CENTOS
package-cleanup
package-cleanup(1) package-cleanup(1)
NAME
package-cleanup - clean up locally installed, duplicate, or orphaned packages
SYNOPSIS
package-cleanup [options] <item ...>
DESCRIPTION
package-cleanup is a program for cleaning up the locally-installed RPMs.
GENERAL OPTIONS
-c <config file>
Use alternative config file (default is /etc/yum.conf).
-h, --help
Help; display a help message and then quit.
-q, --quiet
Print out nothing unnecessary.
-v, --version
Report program version and exit.
-y Agree to anything asked.
--leaves
List leaf nodes in the local RPM database. Leaf nodes are RPMs that are not relied upon by any other RPM.
--orphans
List installed packages which are not available from currently configured repositories. This is identical to "yum list extras",
which may provide better output.
--oldkernels
Remove old kernel and kernel-devel packages.
--problems
List dependency problems in the local RPM database. If any problems are found it will exit with an exit code of 1.
--dupes
Scan for duplicates in the local RPM database.
--cleandupes
Scan for duplicates in the local RPM database and clean out the older versions.
--count <COUNT>
Number of duplicate/kernel packages to keep on the system (default 2)
LEAVES OPTIONS
--all When listing leaf nodes also list leaf nodes that are not libraries.
--leaf-regex
A package name that matches this regular expression will be considered a leaf.
--exclude-devel
When listing leaf nodes do not list development packages.
--exclude-bin
When listing leaf nodes do not list packages with files in bin directories.
OLDKERNELS OPTIONS
--keepdevel
Do not remove kernel-devel packages when removing kernels
EXAMPLES
List all dependency problems:
package-cleanup --problems
List all packages that are not in any Yum repository:
package-cleanup --orphans
Remove old kernels keeping 3 and leaving old kernel-devel packages installed:
package-cleanup --oldkernels --count=3 --keepdevel
List all leaf packages with no files in a bin directory whose name begins with either 'perl' or 'python':
package-cleanup --leaves --exclude-bin --leaf-regex="^(perl)|(python)"
FILES
As package-cleanup uses YUM libraries for retrieving all the information, it relies on YUM configuration for its default values like which
repositories to use. Consult YUM documentation for details:
/etc/yum.conf
/etc/yum/repos.d/
/var/cache/yum/
SEE ALSO
yum.conf (5)
http://yum.baseurl.org/
AUTHORS
See the Authors file included with this program.
BUGS
There are of course no bugs, but should you find any, you should first consult the FAQ section on http://yum.baseurl.org/wiki/Faq and if
unsuccessful in finding a resolution contact the mailing list: yum-devel@lists.baseurl.org. To file a bug use http://bugzilla.redhat.com
for Fedora/RHEL/Centos related bugs and http://yum.baseurl.org/report for all other bugs.
Gijs Hollestelle 03 November 2005 package-cleanup(1)