12-23-2008
Finding duplicates from positioned substring across lines
I have million's of records each containing exactly 50 characters and have to check the uniqueness of 4 character substring of 50 character (postion known prior) and report if any duplicates are found.
Eg. data...
AAAA00000000000000XXXX0000 0000000000... upto50 chars
AAAA00000000000000XXXY0000 0000000000... upto50 chars
AAAA00000000000000XXXY0000 0000000000... upto50 chars
output:
Duplicates are found for XXXY.
I'm new to unix scripting. Can anyone provide me direction?
~GAP
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a huge file (over 30mb) that I am processing through with perl. I am pulling out a list of filenames and placing it in an array called @reports.
I am fine up till here. What I then want to do is go through the array and find any duplicates. If there is a duplicate, output it to the screen.... (3 Replies)
Discussion started by: dangral
3 Replies
2. Shell Programming and Scripting
hii,
i want to know the shell command for finding the last occurance of a substring in string..
i can use grep command or sed to find out the occurance of a substring in a string but how do i find out the last occurance.shud i use grep amd and cut the string everytime and store it in a new... (7 Replies)
Discussion started by: cutelucks
7 Replies
3. Shell Programming and Scripting
I am trying to figure out how to scan a file like so:
1 ralphs office","555-555-5555","ralph@mail.com","www.ralph.com
2 margies office","555-555-5555","ralph@mail.com","www.ralph.com
3 kims office","555-555-5555","kims@mail.com","www.ralph.com
4 tims... (17 Replies)
Discussion started by: totus
17 Replies
4. Shell Programming and Scripting
I will be performing a task on several directories, each containing a large number of files (2500+) that follow a regular naming convention:
YYYY_MM_DD_XX.foo_bar.A.B.some_different_stuff.EXT
What I would like to do is automatically discover the part of the filenames that are common to all... (1 Reply)
Discussion started by: cmcnorgan
1 Replies
5. Shell Programming and Scripting
Input:
a
b
b
c
d
d
I need:
a
c
I know how to get this (the lines that have duplicates) :
b
d
sort file | uniq -d
But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies
6. Shell Programming and Scripting
I am currently creating a script to find filenames that are listed once in an input file (find non duplicates). I then want to report those single files in another file. Here is the function that I have so far:
function dups_filenames
{
file2=""
file1=""
file=""
dn=""
ch=""
pn=""
... (6 Replies)
Discussion started by: chipblah84
6 Replies
7. Shell Programming and Scripting
Hi team,
I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record.
can one help me on finding the duplicates,
Thanks in advance.
... (2 Replies)
Discussion started by: baskivs
2 Replies
8. UNIX for Dummies Questions & Answers
Hi everyone. I'm trying to help my wife with a project, she has exported 200 images from many different folders, unfortunately there was a problem with the export and I need to find the master versions so that she doesn't have to go through and select them again.
I need to:
For each image in... (2 Replies)
Discussion started by: Rhinoskin
2 Replies
9. Shell Programming and Scripting
I have unix file like below
>newuser
newuser
<hello
hello
newone
I want to find the unique values in the file(excluding <,>),so that the out put should be
>newuser
<hello
newone
can any body tell me what is command to get this new file. (7 Replies)
Discussion started by: shiva2985
7 Replies
10. UNIX for Beginners Questions & Answers
I have a text file that has some data like:
PADHOGOA1 IOP055_VINREG5_1 ( .IO(VINREG5_1), .MONI(), .MON_D(px_IOP055_VINREG5_1_MON_D), .R0T(px_IOP054_VINREG5_0_R0T), .IO1() );
PADV30MA0 IOP056_VOUT3_IN ( .IO(VOUT3_IN), .V30M(px_IOP056_VOUT3_IN_V30M));
PADV30MA0 IOP057_VOUT3_OUT (... (2 Replies)
Discussion started by: utkarshkhanna44
2 Replies
textutil::trim(n) Text and string utilities, macro processing textutil::trim(n)
__________________________________________________________________________________________________________________________________________________
NAME
textutil::trim - Procedures to trim strings
SYNOPSIS
package require Tcl 8.2
package require textutil::trim ?0.7?
::textutil::trim::trim string ?regexp?
::textutil::trim::trimleft string ?regexp?
::textutil::trim::trimright string ?regexp?
::textutil::trim::trimPrefix string prefix
::textutil::trim::trimEmptyHeading string
_________________________________________________________________
DESCRIPTION
The package textutil::trim provides commands that trim strings using arbitrary regular expressions.
The complete set of procedures is described below.
::textutil::trim::trim string ?regexp?
Remove in string any leading and trailing substring according to the regular expression regexp and return the result as a new
string. This is done for all lines in the string, that is any substring between 2 newline chars, or between the beginning of the
string and a newline, or between a newline and the end of the string, or, if the string contain no newline, between the beginning
and the end of the string. The regular expression regexp defaults to "[ \t]+".
::textutil::trim::trimleft string ?regexp?
Remove in string any leading substring according to the regular expression regexp and return the result as a new string. This apply
on any line in the string, that is any substring between 2 newline chars, or between the beginning of the string and a newline, or
between a newline and the end of the string, or, if the string contain no newline, between the beginning and the end of the string.
The regular expression regexp defaults to "[ \t]+".
::textutil::trim::trimright string ?regexp?
Remove in string any trailing substring according to the regular expression regexp and return the result as a new string. This apply
on any line in the string, that is any substring between 2 newline chars, or between the beginning of the string and a newline, or
between a newline and the end of the string, or, if the string contain no newline, between the beginning and the end of the string.
The regular expression regexp defaults to "[ \t]+".
::textutil::trim::trimPrefix string prefix
Removes the prefix from the beginning of string and returns the result. The string is left unchanged if it doesn't have prefix at
its beginning.
::textutil::trim::trimEmptyHeading string
Looks for empty lines (including lines consisting of only whitespace) at the beginning of the string and removes it. The modified
string is returned as the result of the command.
BUGS, IDEAS, FEEDBACK
This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category textutil
of the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have for
either package and/or documentation.
SEE ALSO
regexp(n), split(n), string(n)
KEYWORDS
prefix, regular expression, string, trimming
textutil 0.7 textutil::trim(n)